Can't find archive on external local storage #4

YoSiJo · 2019-05-02T16:31:11Z

Nextcloud 15.0.7

I tried to convert a webm file to an mp4. The file itself is located on an external storage that is locally embedded.
Unfortunately I couldn't find anything else in the logs.

PaulLereverend · 2019-05-02T16:43:34Z

Okay, could you try to do the same on local storage ?
Are you the owner of the external storage ?
Are you connected with an admin account ?
What OS is running your NC instance ?

YoSiJo · 2019-05-02T17:38:50Z

Okay, could you try to do the same on local storage?

I tried it in my home directory. There it goes.
I also tried it again in the External Storage (Type: Local) first level and in a second level. In both cases it is missing.
For a better understanding the structure:

/ (Home)
/storage (External Storage, Type: Local, Real path: /mnt/storage)
/storage/videos (External Storage, Type: Local, Real path: /mnt/videos)

Are you the owner of the external storage?

The external storage is shared by the administrator for groups.

Are you connected with an admin account?

Yes

What OS is running your NC instance?

Debian 9.9
PHP 7.2
Storage FS:

Mount: /
Type: btrfs
Mount: /mnt/*
Type: GlusterFS

PaulLereverend · 2019-05-02T18:07:07Z

Thank you, I will try to reproduce the issue in order to fix it. I'll keep you updated.

PaulLereverend · 2019-05-05T18:07:48Z

I was not able to reproduce the issue so I added some details in the error message.
Could you try this version and send me the error please ?
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.3/video_converter.tar.gz
(extract into nextcloud/apps and enable it)

YoSiJo · 2019-05-05T21:43:43Z

Now i see this:

May 05 23:42:07 … Nextcloud[15997]: {PHP} Undefined variable: desc at /var/www/…/apps/video_converter/lib/Controller/ConversionController.php#60

YoSiJo · 2019-05-06T13:11:05Z

That was a quick test in the evening. If I can do more tests, I will be happy to help. But the loglevel is already set to 0.

PaulLereverend · 2019-05-06T20:12:58Z

Thank you very much !
Could you try this version ? I fixed the desc issue.
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.4/video_converter.tar.gz

YoSiJo · 2019-05-07T07:36:17Z

This time I didn't get any message in the log. But the app shows the following:

Error converting 1531172467907.webm

Can't find video on external local storage :

PaulLereverend · 2019-05-07T20:52:28Z

Thank you again for your help ! This means the app is not able to get the external mount points.
I've made a version with many logs on this part to help me figure out where is the problem. You can find it here. The window with the loading animation wont disappear and you should find a few things on your browser console.
You can find this version here : https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.5/video_converter.tar.gz

YoSiJo · 2019-05-08T07:27:00Z

I found this:

User:YoSiJoarray(17) {
  ["/YoSiJo/files/foo/baa"]=>
  array(10) {
    ["mountpoint"]=>
    string(14) "foo/baa"
    ["class"]=>
    string(5) "local"
    ["backend"]=>
    string(5) "Lokal"
    ["authMechanism"]=>
    string(10) "::"
    ["personal"]=>
    bool(false)
    ["options"]=>
    array(1) {
      ["datadir"]=>
      string(33) "/var/www/nextcloud.example.com/mnt/baa"
    }
    ["mountOptions"]=>
    array(6) {
      ["encrypt"]=>
      bool(true)
      ["previews"]=>
      bool(true)
      ["enable_sharing"]=>
      bool(true)
      ["filesystem_check_changes"]=>
      int(1)
      ["encoding_compatibility"]=>
      bool(false)
      ["readonly"]=>
      bool(false)
    }
    ["priority"]=>
    int(150)
    ["applicable"]=>
    array(2) {
      ["groups"]=>
      array(1) {
        [0]=>
        string(12) "fooShare"
      }
      ["users"]=>
      array(1) {
        [0]=>
        string(6) "YoSiJo"
      }
    }
    ["id"]=>
    int(6)
  }
  ["/YoSiJo/files/foo…
conversion.js:147:33

And this:

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data[Weitere Informationen] conversion.js:148:44

PaulLereverend · 2019-05-08T10:14:48Z

I think I found the issue. I used the ["backend"]=> string(5) "Lokal" in order to check the type of external storage but with the English and French spelling (Local) and I thought It was in English for everyone. I now use ["class"]=>string(5) "local" which seems to be the same on all languages.

Can you confirm that this version is working for you ?https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.6/video_converter.tar.gz

YoSiJo · 2019-05-08T11:39:59Z

This time there was the following problem:

Can't find video on external local storage : /var/www/nextcloud.example.com/mnt/dir1/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir4/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir5/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir6/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir7/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir2/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir8/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir9/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir10/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir11/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir12/main/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir13/main/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir14/main/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir15/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir16/main/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir17/main/dir2/dir3/1531172467907.webm not found /var/www/nextcloud.example.com/mnt/dir18/main/dir2/dir3/1531172467907.webm not found

The dir[0-9]+ names come from me. I replaced the original names with them.

PaulLereverend · 2019-05-08T16:02:35Z

Is one of the paths the good one to your video ?

YoSiJo · 2019-05-08T16:17:58Z

Short, no. Correct is: /var/www/nextcloud.example.com/mnt/dir2/dir3/1531172467907.webm
Ok, I recognize that my setup is a bit unfriendly. ;)
I have the following structure:
Nexloud Path: / = Real: /var/www/nextcloud.example.com/data/$USER/files
Nexloud Path: /dir4 = Real: /var/www/nextcloud.example.com/mnt/dir4
Nexloud Path: /dir4/dir2 = Real: /var/www/nextcloud.example.com/mnt/dir2
Nexloud Path: /dir4/dir2/dir3 = Real: /var/www/nextcloud.example.com/mnt/dir2/dir3

Converting videos works in the following NextCloud folders:

/
/dir4

But not in:

/dir4/dir2
/dir4/dir2/dir3

If you want, just send me an email to PaulLereverend@yosijo.de and I'll set up an account in the NextCloud with this kind of structure. Then testing might be easier.

YoSiJo · 2019-05-08T16:33:29Z

I expected this to be posted automatically. Unfortunately not, therefore here the short reference to the Bounty: https://www.bountysource.com/issues/73634686-can-t-find-archive-on-external-local-storage

PaulLereverend · 2019-05-08T16:36:09Z

Okay I think I understood (finally 😄). You have external folder mounted inside another external folder (I did not knew that was possible). I tried and it looks like I have the same issue. I will work on a fix and I will comeback to you in a few minutes.

PaulLereverend · 2019-05-08T17:28:54Z

This version fixed the issue I had when I mount external folders inside external folders. I hope this was the one you had. Could you try this one and tell me if it works for you ?
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.7/video_converter.tar.gz

YoSiJo · 2019-05-08T17:48:00Z

Now it works in the following folders:

/ (/var/www/nextcloud.example.com/data/$USER/failes)
/dir4 (/var/www/nextcloud.example.com/mnt/dir4)
/dir4/dir2 (/var/www/nextcloud.example.com/mnt/dir2)

but not in /dir4/dir2/dir3 (/var/www/nextcloud.example.com/mnt/dir2/dir3)

I beg your pardon for this construct. ^^

Log

Can't find video on external local storage : /var/www/nextcloud.example.com/mnt/dir1/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir4/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir5/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir6/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir7/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir2/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir8/dir2/dir3/1531172467907.mp4 not found /var/www/nextcloud.example.com/mnt/dir9/dir2/dir3/1531172467907.mp4 not found …

PaulLereverend · 2019-05-08T19:42:19Z

I made some changes on this one. It may work for you https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.8/video_converter.tar.gz

YoSiJo · 2019-05-08T20:01:34Z

Well, now he converts everything but from dir2 and dir3 the file always ends up in dir4.

List of Test:

/1531172467907.webm > /1531172467907.avi 
/dir4/1531172467907.webm > /dir4/1531172467907.avi
/dir4/dir2/1531172467907.webm > /dir4/1531172467907.avi
/dir4/dir2/dir3/1531172467907.webm > /dir4/1531172467907.avi

PaulLereverend · 2019-05-08T20:31:15Z

I was aware of that bug. It happens when there is multiple files with the same name on external folders mounted inside another external folder. The good thing is that it means everything else is working. The other good thing is I found a way to fix that. You can find it here :
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.9/video_converter.tar.gz

YoSiJo · 2019-05-08T20:38:25Z

Now I've made that mistake again: Can't find video on external local storage : …
To be on the safe side I downloaded and integrated the file two. I also restarted the apache2.
Do you need the paths?

PaulLereverend · 2019-05-08T20:53:15Z

Did you had the debugger ? (pause before converting).
It should have worked, but if you see that error it means the mtime is wrong somewhere.
In order to see that I added some log on your browser console on that version. You should see one "mtime : xxxxxxxxxxxx" and a few "filemtime : xxxxxxxxxxxxx"
Don't forget to clear the browser cache for this one.
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.10/video_converter.tar.gz

YoSiJo · 2019-05-08T21:12:04Z

This time it loads infinitely and I see the following message in the console:

mtime : 1557349379000 filemtime : 1557349394 filemtime : 1557347664 "{\"code\":0,\"desc\":\"Can't find video on external local storage : \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Wissen\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/dirw\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Anime\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Audio\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/beets\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Videos\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/dirbaa\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Backup\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Distros\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Dokumentationen\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/eBooks\\\/main\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/dirboo\\\/main\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/dirfoo\\\/main\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/CoinPG\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Minecraft\\\/main\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Musik\\\/main\\\/\\\/1531172467907.webm not found \\\/var\\\/www\\\/nextcloud.example.com\\\/mnt\\\/Programme\\\/main\\\/\\\/1531172467907.webm not found \"}" conversion.js:148:33
SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data[Weitere Informationen] conversion.js:149:44
    success https://nextcloud.example.com/apps/video_converter/js/conversion.js?v=164bddff-12:149
    j https://nextcloud.example.com/core/vendor/core.js?v=164bddff-12:2
    fireWith https://nextcloud.example.com/core/vendor/core.js?v=164bddff-12:2
    x https://nextcloud.example.com/core/vendor/core.js?v=164bddff-12:4
    b https://nextcloud.example.com/core/vendor/core.js?v=164bddff-12:4

I clear the cache, restarted apache and tried the whole thing again in the anonymous window.

PaulLereverend · 2019-05-09T20:31:16Z

This output does not make sense. I tried many cases with the exact same structure and everything worked fine. Could you set up a small instance of NC with the exact same structure (dont put personal files) and give me a NC admin account and FTP access to the apps folder. It will be faster and easier for me to debug.
You can send me the credentials to : paulereverend@gmail.com

YoSiJo · 2019-05-11T14:33:58Z

Your account is created. You should have received an e-mail. Just use the password reset function.
I have currently tested again and the error is currently as follows.
It converts the video, but the loading process doesn't end. If you now refresh the page, the video is there.
But it works everywhere now.

PaulLereverend · 2019-05-11T20:18:05Z

Oh with your last message I thought it was not converting. The infinite load is normal, this logs prevent the process from ending normally.
Could you try this clean version ? I will also test it on your demo instance to make sure everything is ok.
https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.11/video_converter.tar.gz

Edit: I can't use the reset the password function, I got the following error
{"status":"error","msg":"Impossible d'envoyer le courriel de r\u00e9initialisation. Veuillez v\u00e9rifier que votre nom d'utilisateur est correct."}

{"status":"error","msg":"Unable to send verification email. Check your username"}

Edit2 : This email has been successfully sent a few minutes later.

YoSiJo · 2019-05-12T12:04:41Z

Password

Ok Thanks for the hint with the password reset function. The error can also be corrected and I also have an error message in the log. I looked to see if I could find an issue for it.

Converting

As far as I can see, the conversion now works in every subfolder.
As far as I would see this issue as fixed.
But what's funny is that the video file under TestGFS0 still has this bug with "Not found". But as soon as I copy the video file, I can convert the copied version everywhere. Only not the original file.
Should I open an extra issue for it or do you want to cover it here?

PaulLereverend · 2019-05-12T12:51:53Z

I will cover that here since it is related to the latest changes.
Yes I saw that aswell and I think it is related to the mtime but i'm still not able to reproduce this issue with all the files I try on my instances. Could you mount the /apps/video_converter folder and give me access so I don't have to send you tons of debug versions.

YoSiJo · 2019-05-12T13:00:53Z

It's done. Should now have access. Remember that I have use OPCache. I don't know if this might cause code updates to be ignored.
Here the stats from file:

stat ../mnt/TestGFS0/main/1531172467907test.webm
  File: ../mnt/TestGFS0/main/1531172467907test.webm
  Size: 1638057         Blocks: 3200       IO Block: 131072 regular file
Device: 7ah/122d        Inode: 11869544217668404032  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2019-05-11 23:17:28.761025334 +0200
Modify: 2019-05-11 22:31:20.188833756 +0200
Change: 2019-05-11 23:17:12.548791499 +0200
 Birth: -

PaulLereverend · 2019-05-12T14:00:21Z

My tests showed that nextcloud thinks the video has been modified 12 days ago, but php and your stats showed that it had been modified yesterday. That's why the app thinks those are not the same videos.
On the external storage settings, could you set check for changes to once every direct access ?

YoSiJo · 2019-05-12T14:42:27Z

Actually, yeah and i have checked that again.
To be on the safe side, I have now started a complete file scan again. It should be done in 3-4h.

PaulLereverend · 2019-05-12T15:16:03Z

I rewrote the file-checking in another way which seems to work on every video on your instance (even the one that did not work previously).
You can confirm that this version solve the issue. https://github.com/PaulLereverend/NextcloudVideo_Converter/releases/download/0.0.12/video_converter.tar.gz

YoSiJo · 2019-05-12T15:56:34Z

That's nice. Everything works for me now.
I would then withdraw your access to the apps or and clear the TestGFS*. You can use the account if you want.
The issue should be fixed, or how do you see it?

PaulLereverend · 2019-05-12T16:20:31Z

I'm happy to hear that !
You can delete my account, thank you for helping me debuging.
This issue is fixed, if you find another bug, feel free to open another ticket

PaulLereverend closed this as completed May 12, 2019

RodSeq mentioned this issue Jul 25, 2020

Issue with VideoConverter v0.1.4 and Nextcloud v18.0.7 #50

Open

Can't find archive on external local storage #4

Can't find archive on external local storage #4

Comments

YoSiJo commented May 2, 2019

PaulLereverend commented May 2, 2019

YoSiJo commented May 2, 2019

Okay, could you try to do the same on local storage?

Are you the owner of the external storage?

Are you connected with an admin account?

What OS is running your NC instance?

PaulLereverend commented May 2, 2019

PaulLereverend commented May 5, 2019

YoSiJo commented May 5, 2019

YoSiJo commented May 6, 2019

PaulLereverend commented May 6, 2019

YoSiJo commented May 7, 2019

PaulLereverend commented May 7, 2019

YoSiJo commented May 8, 2019

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019 • edited Loading

YoSiJo commented May 8, 2019

PaulLereverend commented May 8, 2019

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019

Log

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019

PaulLereverend commented May 8, 2019

YoSiJo commented May 8, 2019

PaulLereverend commented May 9, 2019

YoSiJo commented May 11, 2019

PaulLereverend commented May 11, 2019 • edited Loading

YoSiJo commented May 12, 2019

Password

Converting

PaulLereverend commented May 12, 2019

YoSiJo commented May 12, 2019 • edited Loading

PaulLereverend commented May 12, 2019 • edited Loading

YoSiJo commented May 12, 2019

PaulLereverend commented May 12, 2019

YoSiJo commented May 12, 2019

PaulLereverend commented May 12, 2019

YoSiJo commented May 8, 2019 •

edited

Loading

PaulLereverend commented May 11, 2019 •

edited

Loading

YoSiJo commented May 12, 2019 •

edited

Loading

PaulLereverend commented May 12, 2019 •

edited

Loading