FreeBSD UFS2 file per directory limit

[giraudeo]

Steps to reproduce

1. Install FreeBSD with default partition type (UFS)
2. Install nextcloud
3. Upload your files
4. Generate previews with previewgenerator app

Expected behaviour

All previews should be generated

Actual behaviour

Only first 32765 previews are generated

Server configuration

Operating system: FreeBSD 12

Web server: nginx

Database: MariaDB 10.3

PHP version: 7.1

Nextcloud version: 15.0.0

Updated from an older Nextcloud/ownCloud or fresh install: Updated

Where did you install Nextcloud from: Official repository

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - bruteforcesettings: 1.2.0 - calendar: 1.6.4 - cloud_federation_api: 0.1.0 - comments: 1.5.0 - contacts: 3.0.0 - dav: 1.8.0 - federatedfilesharing: 1.5.0 - federation: 1.5.0 - files: 1.10.0 - files_pdfviewer: 1.4.0 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_trashbin: 1.5.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - gallery: 18.2.0 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - nextcloud_announcements: 1.4.0 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - provisioning_api: 1.5.0 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - theming: 1.6.0 - twofactor_backupcodes: 1.4.1 - updatenotification: 1.5.0 - workflowengine: 1.5.0 Disabled: - admin_audit - encryption - files_accesscontrol - files_antivirus - files_external - user_external - user_ldap

Nextcloud configuration:

Config report

"system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "http:\/\/localhost", "dbtype": "mysql", "version": "15.0.0.10", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "filelocking.enabled": false, "instanceid": "***REMOVED SENSITIVE VALUE***", "auth.bruteforce.protection.enabled": false, "maintenance": false, "theme": "", "loglevel": 0, "updater.release.channel": "stable" }

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Safari

Operating system: MacOS

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"FVn52UwMTvg8lwzAErGz","level":0,"time":"2018-12-15T19:02:31+00:00","remoteAddr":"X.X.X.X","user":"XXX","app":"PHP","method":"GET","url":"/nextcloud/index.php/core/preview.png?file=text.txt&x=281&y=500&a=1&mode=cover","message":"mkdir(): Too many links at /usr/local/www/nextcloud/lib/private/Files/Storage/Local.php#85","userAgent":"Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.7","version":"15.0.0.10"}
{"reqId":"4dvrvLb2Ei1q7J5NUf9H","level":0,"time":"2018-12-15T19:04:35+00:00","remoteAddr":"X.X.X.X","user":"XXX","app":"PHP","method":"GET","url":"/nextcloud/index.php/core/preview.png?file=text.txt&x=281&y=500&a=1&mode=cover","message":"mkdir(): Too many links at /usr/local/www/nextcloud/lib/private/Files/Storage/Local.php#85","userAgent":"Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.7","version":"15.0.0.10"}

[giraudeo]

Hello,

First of all, just a big thanks to all the Nextcloud team for the hard work you are providing to the open-source community. I am using Nextcloud for 1 year and half for personal usage, and I am very happy with it up to now ;)

So, to summarise my bug, I spent a good part of yesterday to see why and where is coming this UFS2 limitation.
Best summary would be this thread:
https://lists.freebsd.org/pipermail/freebsd-fs/2005-March/001001.html

I am not a FS expert, but for what I understand, the default inode table on UFS2 is made to accept only 32767 references.
Different workarounds would be to modify this inode limit, or to move to a different FS (or OS).

Right now, I do not have really this possibility, and was wondering if something could be done on the application level?

For my defence, I found other discussions where it is not recommended to go with large amount of files inside a directory.
like here:
https://forums.freebsd.org/threads/how-many-files-could-be-stored-in-a-directory.48787/

Thx again!

Regards,

Olivier

[tflidd]

Perhaps you could program an app that distributes the files over several folders but shows it as a single folder in the interface. Many discourage to use too many files in a single folder: https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory

Is there still a specific bug in Nextcloud you are referring to? If it's more about how to realize your setup, you might be better off in the forum help.nextcloud.com

[giraudeo]

Hey tflidd,

Thanks for your reply.
The problem is coming from the flat structure of the Nextcloud preview generation itself.

All previews are stored in next cloud/data/appdata_XXXXXX/preview, and it reaches this 32K files limit:

root@localhost:/usr/local/www/nextcloud/data/appdata_XXXXXX/preview # mkdir mytest
mkdir: mytest: Too many links

root@localhost:/usr/local/www/nextcloud/data/appdata_XXXXXX/preview # ls -a | wc -l
32767

Let me know if you need more details.

Regards,

Olivier

[tflidd]

Thanks for clarification. Could perhaps increase the performance on certain systems.

@rullzer

[gbmaster]

The bug is also present in OpenBSD using FFS.

Running
ls -a | wc -l
returns 32767

[skjnldsv]

Closing as wontfix, please increase the number of available inodes. For me this is an issue outside of NC