Unable to export large account #385

prairietree · 2023-03-28T22:58:26Z

Hello, I hope it is going well for you.

I will let you in on a little of my experience with using user migration tool. It sounds like a nice idea but so far it has been only frustration for me. The user I am trying to export has about 27BG of data. I know that is a lot but I was hoping it would work.

The first issue I had was with exporting. It would run a few minutes (probably less then 10 and then say it is done successfully). But nextcloud always said the file was zero KB and would not let me download. I worked through some errors in the log in the admin account but none of them ended up being related. I ran it a few more times while watching the log and watching the file size. Sometimes I deleted the file in between runs. I think now it was just not done when it said it was. Also, I think it was not really a zero byte file. It was just nextcloud that said it was.

I finally let one run for a day or so and the file ended up at about 18GB (I would hope I watched the file size for a while to make sure it was done, but I am not sure if I did). NextCloud still said it was zero byte but it would let me download it. I thought maybe that was good but I decided to try again since it seemed kind of small. On the second run after a while I had a 37GB file. I am not sure if I deleted the file in between runs. That seems kind of large but I moved it over and tried to import it.

I had the same kind of user interface experience. It runs for a few minutes (maybe 5-7) then says "Import completed successfully", but when I look around I do not find any files. In this case there is an item in the log. It says "ValueError: Invalid or uninitialized Zip object". That makes me think it is a bad file.

It is not hard to get more then 27BGs of data but it feels like the migration tool is not made to work with that large of a data set.

I saw someone talk about running it form the command line but was not able to find documentation on that. Could someone point me to that? I will try running it form the command line.

I think it would be nice to have examples of running it from the command line on the readme (https://github.com/nextcloud/user_migration#readme).

Also, it would be nice to have a more detailed report. Like the total number of files imported or exported and the total size. Or just something to help determine if it ran successfully.

I am using "Nextcloud Hub 3 (25.0.4)"

Thanks for your help.

Pytal · 2023-03-29T00:39:45Z

Thanks for the report @prairietree

Please try exporting and importing from the CLI with the respective commands by referring to the output from the occ commands below

occ user:export --help
occ user:import --help

prairietree · 2023-03-29T15:07:05Z

This seems much better but I am not sure what is going on. I ran:

$ sudo -u www-data php /var/www/nextcloud/occ user:export -v user-name /data/ocexport/
Exporting user information in user.json…
Exporting settings in settings.json…
Exporting versions in versions.json…
Exporting calendars into dav/calendars/…
Skipping calendar "Contact birthdays" containing invalid calendar data
Skipping calendar "default" containing invalid calendar data
No calendars to export…
Exporting contacts into dav/address_books/…
No contacts to export…
Exporting trashbin into files_trashbin…
Exporting trashbin files…
Exporting account information in settings/account.json…
Exporting profile config in settings/config.json…
Exporting files…

$ ll -a /data/ocexport/
total 8
drwxr-xr-x 2 www-data www-data 4096 Mar 28 18:22 ./
drwxr-xr-x 5 root     root     4096 Mar 28 18:22 ../

It has been stuck there for over 12 hours now and there is no file in /data/ocexport/. Is it possible it is still running or has it failed?

Thanks

Pytal · 2023-03-29T18:01:10Z

It appears from the stdout that the export command exited while files were being exported

Please run the same command with -vvv for more verbose output

prairietree · 2023-03-29T23:06:48Z

ok, I had some trouble getting the old process killed. I had to reboot for it to go away. I ran the command with three v's but am getting the same results. It did not give me any more output and stops at the same place. I have let it sit at this point a few hours now.

sudo -u www-data php /var/www/nextcloud/occ user:export -vvv user-name /data/ocexport/

Edit:
Rebooted again and tried to run without switching to www-data user. Would not run. I then switched to root and changed to the nextcloud directory and ran sudo -u www-data php occ user:export -vvv ken-lt1 /data/ocexport/. I will leave that one run for a while.

If I would leave off the path, what would be the default location?

Edit two:
Ran again and watched logs at same time and found this:
{"reqId":"pVdjkVEgWoigofzfigct","level":1,"time":"2023-03-29T23:51:40+00:00","remoteAddr":"","user":"--","app":"related_resources","method":"","url":"--","message":"Could not resolve OCA\Circles\CirclesManager! Class OCA\Circles\CirclesManager does not exist","userAgent":"--","version":"25.0.5.1","data":{"app":"related_resources"}}

I think I turned off circles because of another error I was seeing earlier.

Pytal · 2023-03-29T23:30:42Z

The export destination path is required as there is no default location

It seems that the core issue is in the files export, @come-nc how could @prairietree debug this?

prairietree · 2023-03-30T00:02:00Z

Some more items came up in the log that might be useful:

{"reqId":"vfUw2CS9SX3hzr2wGVNq","level":1,"time":"2023-03-29T23:33:51+00:00","remoteAddr":"192.168.0.202","user":"[user-two]","app":"no app in context","method":"GET","url":"/nextcloud/apps/files/","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","version":"25.0.5.1","data":[]}
{"reqId":"vfUw2CS9SX3hzr2wGVNq","level":1,"time":"2023-03-29T23:33:51+00:00","remoteAddr":"192.168.0.202","user":"[user-two]","app":"no app in context","method":"GET","url":"/nextcloud/apps/files/","message":"Deprecated event type for OCA\\Files::loadAdditionalScripts: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","version":"25.0.5.1","data":[]}
{"reqId":"svdbIvCiDZLSkyejT8jZ","level":1,"time":"2023-03-29T23:34:16+00:00","remoteAddr":"192.168.0.202","user":"[user-two]","app":"no app in context","method":"GET","url":"/nextcloud/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_PRIMARY_KEYS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","version":"25.0.5.1","data":[]}
{"reqId":"svdbIvCiDZLSkyejT8jZ","level":1,"time":"2023-03-29T23:34:26+00:00","remoteAddr":"192.168.0.202","user":"[user-two]","app":"no app in context","method":"GET","url":"/nextcloud/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_INDEXES: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","version":"25.0.5.1","data":[]}
{"reqId":"svdbIvCiDZLSkyejT8jZ","level":1,"time":"2023-03-29T23:34:26+00:00","remoteAddr":"192.168.0.202","user":"[user-two]","app":"no app in context","method":"GET","url":"/nextcloud/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_COLUMNS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","version":"25.0.5.1","data":[]}
{"reqId":"iJoq6GdY0r5OhUQhXGS7","level":3,"time":"2023-03-29T23:36:17+00:00","remoteAddr":"192.168.0.202","user":"[user-name]","app":"no app in context","method":"GET","url":"/nextcloud/ocs/v2.php/apps/notifications/api/v2/notifications?format=json","message":"/[user-name]/files/user.nextcloud_export","userAgent":"Mozilla/5.0 (Linux) mirall/3.4.1 (build 8597) (Nextcloud, ubuntu-5.15.0-69-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCP\\Files\\NotFoundException","Message":"/[user-name]/files/user.nextcloud_export","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Node/LazyUserFolder.php","line":59,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":232,"function":"get","class":"OC\\Files\\Node\\LazyUserFolder","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":121,"function":"getExportFile","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":74,"function":"handleExportDone","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/lib/private/Notification/Manager.php","line":368,"function":"prepare","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/notifications/lib/Controller/EndpointController.php","line":111,"function":"prepare","class":"OC\\Notification\\Manager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"listNotifications","class":"OCA\\Notifications\\Controller\\EndpointController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Node/Root.php","Line":209,"CustomMessage":"--"}}
{"reqId":"T6MetBFafGKpmjTN7sTj","level":3,"time":"2023-03-29T23:41:17+00:00","remoteAddr":"192.168.0.202","user":"[user-name]","app":"no app in context","method":"GET","url":"/nextcloud/ocs/v2.php/apps/notifications/api/v2/notifications?format=json","message":"/[user-name]/files/user.nextcloud_export","userAgent":"Mozilla/5.0 (Linux) mirall/3.4.1 (build 8597) (Nextcloud, ubuntu-5.15.0-69-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCP\\Files\\NotFoundException","Message":"/[user-name]/files/user.nextcloud_export","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Node/LazyUserFolder.php","line":59,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":232,"function":"get","class":"OC\\Files\\Node\\LazyUserFolder","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":121,"function":"getExportFile","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":74,"function":"handleExportDone","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/lib/private/Notification/Manager.php","line":368,"function":"prepare","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/notifications/lib/Controller/EndpointController.php","line":111,"function":"prepare","class":"OC\\Notification\\Manager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"listNotifications","class":"OCA\\Notifications\\Controller\\EndpointController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Node/Root.php","Line":209,"CustomMessage":"--"}}
{"reqId":"nQSsjEsOBJ2zZitl1iUz","level":3,"time":"2023-03-29T23:51:39+00:00","remoteAddr":"192.168.0.202","user":"[user-name]","app":"no app in context","method":"GET","url":"/nextcloud/ocs/v2.php/apps/notifications/api/v2/notifications?format=json","message":"/[user-name]/files/user.nextcloud_export","userAgent":"Mozilla/5.0 (Linux) mirall/3.4.1 (build 8597) (Nextcloud, ubuntu-5.15.0-69-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCP\\Files\\NotFoundException","Message":"/[user-name]/files/user.nextcloud_export","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Node/LazyUserFolder.php","line":59,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":232,"function":"get","class":"OC\\Files\\Node\\LazyUserFolder","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":121,"function":"getExportFile","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":74,"function":"handleExportDone","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/lib/private/Notification/Manager.php","line":368,"function":"prepare","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/notifications/lib/Controller/EndpointController.php","line":111,"function":"prepare","class":"OC\\Notification\\Manager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"listNotifications","class":"OCA\\Notifications\\Controller\\EndpointController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Node/Root.php","Line":209,"CustomMessage":"--"}}
{"reqId":"pVdjkVEgWoigofzfigct","level":1,"time":"2023-03-29T23:51:40+00:00","remoteAddr":"","user":"--","app":"related_resources","method":"","url":"--","message":"Could not resolve OCA\\Circles\\CirclesManager! Class OCA\\Circles\\CirclesManager does not exist","userAgent":"--","version":"25.0.5.1","data":{"app":"related_resources"}}
{"reqId":"0DsoaHHfFJye1ehtHJS2","level":3,"time":"2023-03-29T23:56:17+00:00","remoteAddr":"192.168.0.202","user":"[user-name]","app":"no app in context","method":"GET","url":"/nextcloud/ocs/v2.php/apps/notifications/api/v2/notifications?format=json","message":"/[user-name]/files/user.nextcloud_export","userAgent":"Mozilla/5.0 (Linux) mirall/3.4.1 (build 8597) (Nextcloud, ubuntu-5.15.0-69-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCP\\Files\\NotFoundException","Message":"/[user-name]/files/user.nextcloud_export","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Node/LazyUserFolder.php","line":59,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":232,"function":"get","class":"OC\\Files\\Node\\LazyUserFolder","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":121,"function":"getExportFile","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/user_migration/lib/Notification/Notifier.php","line":74,"function":"handleExportDone","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/lib/private/Notification/Manager.php","line":368,"function":"prepare","class":"OCA\\UserMigration\\Notification\\Notifier","type":"->"},{"file":"/var/www/nextcloud/apps/notifications/lib/Controller/EndpointController.php","line":111,"function":"prepare","class":"OC\\Notification\\Manager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"listNotifications","class":"OCA\\Notifications\\Controller\\EndpointController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Node/Root.php","Line":209,"CustomMessage":"--"}}

come-nc · 2023-03-30T12:30:50Z

You need to let the command run until it’s finished.

@Pytal But regarding the web UI we should look into it this is not the first report saying the UI says import complete before it actually is, this is unexpected.
@prairietree Are you sure about "It would run a few minutes (probably less then 10 and then say it is done successfully)" ? Did you not confuse the message saying the import was planned with one that would say the export is done?
It is not suprising for the file to appear 0 bytes until the export is finished, I think the size is updated only on final write.

The error you get in the logs are suggesting there is a notification about a finished export put the exported file is not actually there, maybe you just deleted it in you testing. We should catch this error and handle it better in our Notifier though.

prairietree · 2023-03-30T13:51:31Z

Hello @come-nc
I deleted through the web UI the first export that failed to import. That is the only thing I have deleted. The one it seems to not find is at "/[user-name]/files/user.nextcloud_export". As an absolute path that path and file does not exits on the system. If is trying to write a temporary file it will probably not have permission.

If that path is suppose to be relative to the NextCloud data directory then it would be the path to the bad export that I deleted. I did not want that big file to be included in the next export I did. But why does it have an issue with it not being there? If it is not there then preferably just do not include it in the export.

Edit:
I did two more test:
sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=account,files user-name /data/ocexport/
Had the same issue.
sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=none user-name /data/ocexport/
Ran successfully.

these test that fail lock up my screen with a process that I cannot kill. The only way to get out is open a new terminal and rebooting the system. It would be much nicer if it exited with an error.

The only other thing I can think of trying is running occ files:scan. Is that a good option?

come-nc · 2023-03-30T14:23:02Z

Hello @come-nc I deleted through the web UI the first export that failed to import. That is the only thing I have deleted. The one it seems to not find is at "/[user-name]/files/user.nextcloud_export". As an absolute path that path and file does not exits on the system. If is trying to write a temporary file it will probably not have permission.

You have a notification in the database that refers to the file. The file does not exists so the render of the notification crashes. If you have access to notifications just dismiss them.

If that path is suppose to be relative to the NextCloud data directory then it would be the path to the bad export that I deleted. I did not want that big file to be included in the next export I did. But why does it have an issue with it not being there? If it is not there then preferably just do not include it in the export.

I do not know if it is relative or not. But it is not the export process but the notification system which runs into the error.
Also, you do not need to delete the export, it will not be included in the next export. This is part of why the name of the file is fixed, so that it can easily be excluded from other exports.

Edit: I did two more test: sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=account,files user-name /data/ocexport/ Had the same issue.

Which issue?

sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=none user-name /data/ocexport/ Ran successfully.

these test that fail lock up my screen with a process that I cannot kill. The only way to get out is open a new terminal and rebooting the system. It would be much nicer if it exited with an error.

So they are not failing, just running. Did you ever see an error message?
How long did you wait before killing them? Can you try letting one run for a long time?

prairietree · 2023-03-30T16:09:18Z

Hello @come-nc I deleted through the web UI the first export that failed to import. That is the only thing I have deleted. The one it seems to not find is at "/[user-name]/files/user.nextcloud_export". As an absolute path that path and file does not exits on the system. If is trying to write a temporary file it will probably not have permission.

You have a notification in the database that refers to the file. The file does not exists so the render of the notification crashes. If you have access to notifications just dismiss them.

I am not sure about the notifications. I have not seen them or had the option to dismiss them. Where would I find them?

If that path is suppose to be relative to the NextCloud data directory then it would be the path to the bad export that I deleted. I did not want that big file to be included in the next export I did. But why does it have an issue with it not being there? If it is not there then preferably just do not include it in the export.

I do not know if it is relative or not. But it is not the export process but the notification system which runs into the error. Also, you do not need to delete the export, it will not be included in the next export. This is part of why the name of the file is fixed, so that it can easily be excluded from other exports.

Good to know.

Edit: I did two more test: sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=account,files user-name /data/ocexport/ Had the same issue.

Which issue?

Same one as described above. Where it gets to the line 'Exporting files…' and just stops and does not return to the prompt.

sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=none user-name /data/ocexport/ Ran successfully.

these test that fail lock up my screen with a process that I cannot kill. The only way to get out is open a new terminal and rebooting the system. It would be much nicer if it exited with an error.

So they are not failing, just running. Did you ever see an error message? How long did you wait before killing them? Can you try letting one run for a long time?

That is the confusing part. I do not get an error message other then maybe in the log file. It just gets to the 'Exporting files…' and stops there and nothing seems to be happening. The first few runs I let run more then 14 hours. The last few I have only let run an hour or so. I never even begins to create an export file in the location I give it. (before that I did some runs from the web UI and they did create files)

prairietree · 2023-04-01T00:23:18Z

I ran sudo -u www-data php /var/www/nextcloud/occ files:scan user-name. It took 4:41 minutes to run. I than ran sudo -u www-data php /var/www/nextcloud/occ user:export -vvv --types=account,files user-name /data/ocexport/ again. It has been running for 26+ hours now. I am thinking it is stuck again.

I am out of ideas and need to get this done. Any ideas on how to move a single user manually?

come-nc · 2023-04-03T09:09:22Z

Which issue?

Same one as described above. Where it gets to the line 'Exporting files…' and just stops and does not return to the prompt.

So they are not failing, just running. Did you ever see an error message? How long did you wait before killing them? Can you try letting one run for a long time?

That is the confusing part. I do not get an error message other then maybe in the log file. It just gets to the 'Exporting files…' and stops there and nothing seems to be happening. The first few runs I let run more then 14 hours. The last few I have only let run an hour or so. I never even begins to create an export file in the location I give it. (before that I did some runs from the web UI and they did create files)

I’m pretty sure it’s just slow and exporting files. Try to let it run for several days. Maybe empty trashbin or delete old versions before if you can to lighten the load.

If it is just too slow, you can migrate the user without the files migrator (should be fast), and migrate the files through any other mean, Nextcloud client sync, scp, sftp, …
That means you will lose trashbin, versions and comments on files. But that may be the best trade off and is what is recommended when there is too much data. (But I agree that 27G is not that much, I would be interested into how long the command takes if you let it run for days and how big is the result, and if you manage to open it how much is from trashbin or versions or such)

prairietree · 2023-04-03T14:17:12Z

@come-nc Thanks for the reply. I did try a little longer run. I let one run for 1 and a half days or about 36 hours. Is there any way I can tell if it is doing something?

I monitor with top and it looks like not much is going on. Mysql and apache were both using about 1-3% cpu. That may be a little more but not much different then they use most of the time. I also looked to see if it is making a file but I never does.

I will look into emptying the trash bin, but part of the reason for using the migration tool was to get old versions of documents. There is not much else I want to migrate over. No settings or shares or data from other apps.

Is there some combination of manually copying over the files and export tool without files that would get me the document histories?

Thanks for your help.

come-nc · 2023-04-04T08:03:38Z

You do not see the file growing on local storage while it’s running?

Maybe you can add directly in user_migration/lib/ExportDestination.php copyFolder method a debug output like echo "Node: ".$node->getName()."\n" in the foreach to see the files listed as they get processed. That will break user_migration UI, it will only work for cli usage.

prairietree · 2023-04-04T16:55:13Z

Thanks for the help @come-nc .

That was a helpful suggestion. I put that in the code and I can see it processing the files now.

I was not seeing any file growing or otherwise in the path I gave it. However, looking at the code it looks like it makes file in the /tmp directory.

In my case that could be an issue because /tmp is on the system drive and only has 33GB of free space. I have another data drive with lots of free space. But I will let it run. If it only uses the 27.6GB that NextCloud says the user is using then it might work.

In case this does not work, would I just edit line 59 ($stream = fopen('php://temp', 'r+');" to open a stream in a different path?

come-nc · 2023-04-05T07:07:12Z

No this temp file is only for the content of one file, temporarily. The ZipStreamer library needs a stream in input, not a string, so when we need to add a string we pass it through a temporary stream. It should not fill /tmp I think, and it is only used for json files, not for exported files themselves.

prairietree · 2023-04-05T12:48:51Z

@come-nc I believe you are right about that. Not sure what I was thinking. However, the /tmp does seem to be getting filled up with a file called 'oc_tmp_K1MM0c-.zip'. It is up to 25GB now.

come-nc · 2023-04-05T13:13:09Z

Indeed, the command exports in /tmp and then moves to the destination, I missed that.
This is in TempExportDestination.php.
We should try to write directly in destination folder I think, to avoid doing a move afterwards, what do you think @Pytal ?

Pytal · 2023-04-06T01:12:23Z

Export archive is moved from /tmp to the specified folder at

user_migration/lib/Command/Export.php

Line 209 in 6b19754

if (rename($path, $folder.'/'.$exportName.'.zip') === false) {

IIRC we had a reason for doing it this way using https://github.com/nextcloud/server/blob/03e965a513accef64015ac307f4a0784f2ffc52d/lib/private/TempManager.php#L79-L114, probably so that failed and incomplete export archives automatically get cleaned up by the OS @come-nc?

prairietree · 2023-04-07T15:15:51Z

@Pytal could it be changed to use the same folder for the temp file but just use a .temp or .part file name tell it is done and then move it to the final file name?

Pytal · 2023-04-21T01:12:36Z

@Pytal could it be changed to use the same folder for the temp file but just use a .temp or .part file name tell it is done and then move it to the final file name?

Sound good technically @come-nc? Also do you remember the original reason for exporting to /tmp first then moving it to the final destination?

come-nc · 2023-04-24T07:18:26Z

@Pytal could it be changed to use the same folder for the temp file but just use a .temp or .part file name tell it is done and then move it to the final file name?

Sound good technically @come-nc? Also do you remember the original reason for exporting to /tmp first then moving it to the final destination?

Yes it sounds good, I do not remember why it was done like that.

Pytal · 2023-04-29T02:55:52Z

@Pytal could it be changed to use the same folder for the temp file but just use a .temp or .part file name tell it is done and then move it to the final file name?

Sound good technically @come-nc? Also do you remember the original reason for exporting to /tmp first then moving it to the final destination?

Yes it sounds good, I do not remember why it was done like that.

Done in #402

come-nc changed the title ~~Unable to import larg account~~ Unable to import large account Mar 30, 2023

come-nc changed the title ~~Unable to import large account~~ Unable to export large account Mar 30, 2023

Pytal mentioned this issue Apr 1, 2023

Listen to notifications for success or failure of export/import #125

Open

2 tasks

This comment was marked as outdated.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to export large account #385

Unable to export large account #385

prairietree commented Mar 28, 2023

Pytal commented Mar 29, 2023

prairietree commented Mar 29, 2023

Pytal commented Mar 29, 2023

prairietree commented Mar 29, 2023 •

edited

Loading

Pytal commented Mar 29, 2023

prairietree commented Mar 30, 2023

come-nc commented Mar 30, 2023

prairietree commented Mar 30, 2023 •

edited

Loading

come-nc commented Mar 30, 2023

prairietree commented Mar 30, 2023

prairietree commented Apr 1, 2023 •

edited

Loading

come-nc commented Apr 3, 2023

prairietree commented Apr 3, 2023

come-nc commented Apr 4, 2023

prairietree commented Apr 4, 2023

come-nc commented Apr 5, 2023

prairietree commented Apr 5, 2023

come-nc commented Apr 5, 2023

Pytal commented Apr 6, 2023

prairietree commented Apr 7, 2023

This comment was marked as outdated.

Pytal commented Apr 21, 2023

come-nc commented Apr 24, 2023

Pytal commented Apr 29, 2023

Unable to export large account #385

Unable to export large account #385

Comments

prairietree commented Mar 28, 2023

Pytal commented Mar 29, 2023

prairietree commented Mar 29, 2023

Pytal commented Mar 29, 2023

prairietree commented Mar 29, 2023 • edited Loading

Pytal commented Mar 29, 2023

prairietree commented Mar 30, 2023

come-nc commented Mar 30, 2023

prairietree commented Mar 30, 2023 • edited Loading

come-nc commented Mar 30, 2023

prairietree commented Mar 30, 2023

prairietree commented Apr 1, 2023 • edited Loading

come-nc commented Apr 3, 2023

prairietree commented Apr 3, 2023

come-nc commented Apr 4, 2023

prairietree commented Apr 4, 2023

come-nc commented Apr 5, 2023

prairietree commented Apr 5, 2023

come-nc commented Apr 5, 2023

Pytal commented Apr 6, 2023

prairietree commented Apr 7, 2023

This comment was marked as outdated.

Pytal commented Apr 21, 2023

come-nc commented Apr 24, 2023

Pytal commented Apr 29, 2023

prairietree commented Mar 29, 2023 •

edited

Loading

prairietree commented Mar 30, 2023 •

edited

Loading

prairietree commented Apr 1, 2023 •

edited

Loading