[Feature]: Upload large files in chunks

[Eidenz]

Feature detail

Currently, uploading is done in a very simple way: upload the whole file directly to the server. However, some users may have limits on their reverse proxy preventing uploading such files.

Uploading in chunks (like LibrePhotos does for exemple) would allow bypassing this problem.

With immich not being able to scan local folders, I think this feature might be needed.

Platform

Server

[bo0tzz]

It probably can't hurt to add the chunking, but I think issues like this should generally be solved on the proxy side. As for watching a folder, we're planning to add an option for that to the cli.

[TheLonelinessOfHS]

This should be implemented in Immich or otherwise, it will result in 1. Additional configuration on the reverse proxy, which I consider as dirty work 2. You cannot increase the size limit of Cloudflare, which is widely used by many people.

[SandiyosDev]

The only way for Cloudflare users proxying their Immich instance to bypass upload limits is by implementing chunking within Immich.

[zackpollard]

I think this has extra benefits besides just the reverse proxy side of things. If we have this, then uploading large files on slow or unstable connections would mean they could be resumed later rather than the entire upload failing. Temporary files could be kept around for a configurable amount of time and if the file continues to be uploaded, negotiation between client and server could be handled so that the client resumes from where it was before.

[fyfrey]

For reliable upload we could implement the tus protocol https://tus.io/

[JVT038]

Additionally, chunked uploading would solve an issue for Cloudflare users.
Cloudflare only allows a max upload size of 100MB. Using chunked uploading, large files can still be uploaded and solve the Cloudflare issue.
Currently, when uploading a large file, it shows this error in the logcat:
flutter : ERROR backupAsset: SocketException: Broken pipe (OS Error: Broken pipe, errno = 32), address = immich.mydomain.com, port = 49742 and the file does not get uploaded.

[perryizgr8]

Version 1.52.0 has my PR: #2101

This solves the CloudFlare tunnel problem for me. I can upload ~ 200 MB files now.

[Thinkscape]

#2101 (comment)

[hungbm]

Version 1.52.0 has my PR: #2101

This solves the CloudFlare tunnel problem for me. I can upload ~ 200 MB files now.

I believe that because of Cloudflare cache. They said the cache is about 512MB, and for my understanding, the big file got splits into many part that under 100mb (cf tunnel allow that) then got re-assemble in CF cache. If total under 512MB => fine, CF will send it to the immich server to save. If larger than 512MB => CF reject, nothing send to Immich selfhost server.

[Slevin7]

Version 1.52.0 has my PR: #2101
This solves the CloudFlare tunnel problem for me. I can upload ~ 200 MB files now.

I believe that because of Cloudflare cache. They said the cache is about 512MB, and for my understanding, the big file got splits into many part that under 100mb (cf tunnel allow that) then got re-assemble in CF cache. If total under 512MB => fine, CF will send it to the immich server to save. If larger than 512MB => CF reject, nothing send to Immich selfhost server.

Thank you for mentioning the cache.
I had the problem for a few days that I kept uploading the same file.
Switching to “developer mode” in Cloudflare helped.

[AdamGoodApp]

Any update if we can get chunking working via web? The current solution for mobile which uses req.headers["Transfer-Encoding"] = "chunked"; does not work for web as Transfer-Encoding header is not allowed to be set in browsers.

[fabiannagel]

Are you sure? Seems like only HTTP/2 has banned it for a better alternative: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

edit: Alright, I see what you mean: https://developer.mozilla.org/en-US/docs/Glossary/Forbidden_header_name

[anh-chu]

Would love to see proper chunked upload since my reverse proxy server is memory/space limited. This means that without chunking, a 400MB file will simply over-utilize the resources of the server, leading to crashes.

[sveken]

Chunking would also allow the use of cloudflare tunnels

[traktuner]

+1 for chunking.
I have fast connections on mobile and on the backend immich server, so uploads from mobile are very fast - on mobile I take 4k videos which have 100+ MB
I would like to put Cloudflare in front of everything because I see some security benefits with that. But the 100 MB Cloudflare limit and non-chunked uploads are not allowing that currently...

[YapWC]

I have tried with Immich App on IOS, on the phone app I believe there is chunking process that helped me with uploading 100+ MB video file with CloudFlare Tunnel (100MB upload limit) as reverse proxy. As of now uploading it directly via the WebUI would still fail but I guess Immich App can act as a temporary solution.

[Hely0n]

You mean the app (and thus the server) already implements chunking?

[Hely0n]

never mind. I just found out that it's not real chunking and that the method of the mobile app won't work in browsers

[hungbm]

I have tried with Immich App on IOS, on the phone app I believe there is chunking process that helped me with uploading 100+ MB video file with CloudFlare Tunnel (100MB upload limit) as reverse proxy. As of now uploading it directly via the WebUI would still fail but I guess Immich App can act as a temporary solution.

I believe that because of Cloudflare cache. They said the cache is about 512MB, and for my understanding, the big file got splits into many part that under 100mb (cf tunnel allow that) then got re-assemble in CF cache. If total under 512MB => fine, CF will send it to the immich server to save. If larger than 512MB => CF reject, nothing send to Immich selfhost server.

[meidlinga]

Eventually this could be implemented by integrating tusd. It would also support pausing and resuming uploads.

[bo0tzz]

Folks, if you don't have anything new to add please just upvote the original request.

[jkjustjoshing]

You can also subscribe to this discussion in the sidebar (or at the bottom of the page on mobile) if you want to be notified when there are updates.

[fabiannagel]

I gave tusd a closer look at it seems pretty straightforward to me (at first glance). We'd probably have the frontend use tus-js-client to talk to tusd in another Docker container. Once uploads are completed, there are event-based hooks we can use to notify Immich's backend about a new file. I'll try to dig some more this weekend.

Bonus: uppy.io natively supports tus so we'd be able to build a rich batch-upload UI for the web.

@alextran1502 What's your take on potential future PR here?

[bo0tzz]

tusd in another Docker container

We've been working on decreasing the number of containers in the stack, so this is probably a no-go. However, tus-node-server can be included in the current server as a library.

Another small caveat with tus is that while there is a dart client, it seems somewhat unmaintained. I don't know enough about flutter/dart or tus to evaluate whether that will give problems though.

I think a PR would be welcome, though I would suggest discussing the design more in-depth first.

[fabiannagel]

Good point regarding Flutter support. I suppose you were talking about jjmutumi/tus_client which is mentioned in the tus docs? There's also a fork with some more recent activity and a Flutter plugin that wraps native iOS/Android implementations.

So maybe the first step could be for someone that actually knows Dart/Flutter to have a look-see at those libraries. Anyone?

[fabiannagel]

A follow-up question, since there's no activity here: Could it be an alternative to use platform-specific libraries within Flutter? I'm talking about the official Android & iOS implementations.

[alextran1502]

@fabiannagel actually on my list of things to do. TUS might clash with background backup

[Dmitrium12]

I'm sorry, do I understand correctly that there is still no solution to this problem? And yes, I'm another cloudflare user.

[Thinkscape]

No solution until chunked upload is implemented in immich (i.e. using uppy+tus)

[aortmannm]

any date or version in which the chunk upload is planned?

[DatNg-J]

Really waiting on this one feature. Is there any timeframe? Or a response about this?

[soccer1902]

A work around is to use the cloudflare wrap app on devices and connect directly to the private ips.

[TheColorman]

@MadeInPierre sounds like it might be running out of memory. Does it also happen when connecting directly to the server?

[MadeInPierre]

@TheColorman Thanks for your reply. Everything is perfect locally, the thumbnails/photos/videos stop loading after some time when going through cloudflare tunnel only. I can't hit the IP directly to test (server is on my RPi at home, exposed through cloudflare only).

[TheColorman]

I just can't imagine it being a cloudflare issue, the browser freezing sounds like an "out of memory" issue on the browser side.

[MadeInPierre]

Nevermind, turns out it works beautifully only when I'm not accessing it from my home network for some reason. Thanks for the answers!

[dasunsrule32]

I have been seeing this exact behavior. I had to pull out my instance from behind Cloudflare as a result because I didn't want to have to different URI's to access Immich. It works fine when accessing it direct over the WAN and on the LAN when I go directly through my router (UDM Pro), but as soon as I route through Cloudflare, whether a tunnel or just proxy it, it disconnects websockets, videos stutter, and long load times when access the application from on-prem through Cloudflare. I don't have this issue with any other applications I self-host through Cloudflare. Not sure what exactly is happening there.

[fraddy91]

I don't know why and how it worked so long, but starting September after update to version 1.114 uploading for files over 100mb stopped working over Couldflare. According to Cloudflare traffic statistics and ddos monitor switching to dns-only mode is not the best solution for little home servers nowadays. So it seems switching to VPN uploading is nearly the best solution for now, isn't it?

[cfelicio]

I’m facing the same issue with uploads over 100MB. All of them are videos, as my photos don’t have enough data to exceed 100MB. I use a Cloudflare tunnel with a reverse proxy. Since I’m only uploading to Immich over a Wi-Fi connection, it would be nice if the app could use a local IP when I’m connected to my home Wi-Fi.

you can do that by setting up a local DNS entry on your router or device that manages DNS for you (e.g. PiHole).

I tried setting that up in my local AdGuard, but I couldn't get it running at 100%. I'm using my default Cloudflare domain for Immich, photos.xxxxx.com, with HTTPS. When I add a local DNS entry pointing photos.xxxxx.com to the local Immich IP (192.168.12.4), it doesn't work because it's expecting a secure endpoint. Did you manage to get this working with Pi-hole?

yes, but I'm using a reverse proxy in front of Immich to provide a valid SSL certificate on the internal network as well. If you are relying on Cloudflare to generate the cert, that might be the issue.

[dasunsrule32]

I’m facing the same issue with uploads over 100MB. All of them are videos, as my photos don’t have enough data to exceed 100MB. I use a Cloudflare tunnel with a reverse proxy. Since I’m only uploading to Immich over a Wi-Fi connection, it would be nice if the app could use a local IP when I’m connected to my home Wi-Fi.

you can do that by setting up a local DNS entry on your router or device that manages DNS for you (e.g. PiHole).

I tried setting that up in my local AdGuard, but I couldn't get it running at 100%. I'm using my default Cloudflare domain for Immich, photos.xxxxx.com, with HTTPS. When I add a local DNS entry pointing photos.xxxxx.com to the local Immich IP (192.168.12.4), it doesn't work because it's expecting a secure endpoint. Did you manage to get this working with Pi-hole?

You have a reverse proxy setup on-prem with a valid cert? That is where you would need to point your A record too.

[m3tc0n]

I’m facing the same issue with uploads over 100MB. All of them are videos, as my photos don’t have enough data to exceed 100MB. I use a Cloudflare tunnel with a reverse proxy. Since I’m only uploading to Immich over a Wi-Fi connection, it would be nice if the app could use a local IP when I’m connected to my home Wi-Fi.

you can do that by setting up a local DNS entry on your router or device that manages DNS for you (e.g. PiHole).

I tried setting that up in my local AdGuard, but I couldn't get it running at 100%. I'm using my default Cloudflare domain for Immich, photos.xxxxx.com, with HTTPS. When I add a local DNS entry pointing photos.xxxxx.com to the local Immich IP (192.168.12.4), it doesn't work because it's expecting a secure endpoint. Did you manage to get this working with Pi-hole?

I had the same problem with Pihole + Unbound. Are you using Unbound? If so, you also have to make A records in the Unbound config.

[Yavieh]

I’m facing the same issue with uploads over 100MB. All of them are videos, as my photos don’t have enough data to exceed 100MB. I use a Cloudflare tunnel with a reverse proxy. Since I’m only uploading to Immich over a Wi-Fi connection, it would be nice if the app could use a local IP when I’m connected to my home Wi-Fi.

you can do that by setting up a local DNS entry on your router or device that manages DNS for you (e.g. PiHole).

I tried setting that up in my local AdGuard, but I couldn't get it running at 100%. I'm using my default Cloudflare domain for Immich, photos.xxxxx.com, with HTTPS. When I add a local DNS entry pointing photos.xxxxx.com to the local Immich IP (192.168.12.4), it doesn't work because it's expecting a secure endpoint. Did you manage to get this working with Pi-hole?

You have a reverse proxy setup on-prem with a valid cert? That is where you would need to point your A record too.

Ah, okay. I’m running an Nginx reverse proxy in my Unraid setup, which handles all the SSL certificate stuff for me. So, in my DNS record, I need to point photos.xxxxxx.com to the local IP of my reverse proxy. Am I understanding this correctly?

[PovilasID]

I set my router I set DNS to point from my domain to the IP of the server and traefik reverse proxy correctly handles however you need to:
a) Have a server with a static IP
b) Know the IP

If you have an OpenWRT router under DHCP and DNS" in "Addresses" subsection you can add /example.com/1.2.3.4 this is effectively an A record

[Compukid]

I dont understand, I also use cloudflare and setup immich behind a reverse proxy like so many. It has been 2 years since this initial conversation and I am having the same issue. I am still doing research and my hope is this is indeed fixed. Otherwise I may need to find another way to self host videos that will support cloudflare and the 100mb limit.

[Yavieh]

I’m also still trying to get it running with local dns redirect, but without luck.
I’m using adguard instead of pihole, but there is also the option to make a dns redirect to a local up without port.
But I’m a little bit confused with your setup. When you create a dns redirect to the exact Immich ip, why also create a nginx proxy host?
Is that used for the outside connection ? But then why no SSL ?
Thanks for you help

[austinvaness]

If you use SSL externally you might need to set up SSL with something like Lets Encrypt to get this working locally. Thats why you probably still need a proxy locally. If you dont, you will likely get errors when you try to connect (especially if you have HSTS enabled). I had to set up a second reverse proxy container because my existing reverse proxy setup for external connections responds to port 4443 on the local network instead of port 443.

[Edzilla2000]

Could we please move those tutorials/discussions elsewhere? They don't contribute to this feature request and are adding a lot of noise.
Please create a separate discussion somewhere with a link that can be posted here for anyone stumbling on this feature request.

[camafive]

I’m also still trying to get it running with local dns redirect, but without luck. I’m using adguard instead of pihole, but there is also the option to make a dns redirect to a local up without port. But I’m a little bit confused with your setup. When you create a dns redirect to the exact Immich ip, why also create a nginx proxy host? Is that used for the outside connection ? But then why no SSL ? Thanks for you help

1. You put your local ip (immich ip), you dont put port because you can't put it there, that is what the reverse proxy is going to do.

2. No, that is not for the outside connection, is just to redirect the traffic to the correct port.
   No SSL because you dont need it locally but you have to set it up for outside connections. I assume you already set that up

[Yavieh]

#13175
@camafive Lets continue talk about that setup here pls. I got still a lot of questions as your setup does not really make sense to me.

[AFumi39]

I think that a way to "mitigate" the problem of upload size limit of Cloudflare could be separate the upload of videos and images.
I mean that the App could allow you to specify when to upload images and when videos. I mean, I want to upload my photos when I take them, under any network. But the videos can be uploaded only under wifi coverage, because trying to upload them under mobile connection can be very slow (even if <100MB) and maybe use much data and battery.
If we can choose upload conditions differently from the 2 types of files, it can help a lot.

Sure, this does not resolve the problem at all, but it helps people that already have Cloudflare for external connection, and nginx when at home.

[SandiyosDev]

This is a pretty good idea, but if chunking is done (cuz some source images can be VERY large), then we won't have to consider special filetypes.

[Daniel-dev22]

How does filebrowser chunk uploads? Other applications do this as well.

[AFumi39]

This is a pretty good idea, but if chunking is done (cuz some source images can be VERY large), then we won't have to consider special filetypes.

Sure, but since we've been waiting chunking for 2 years, and as now there's no official info about its implementation, maybe this workaround con be easier implemented 😄

[raptr55]

one could implement a feature similar to home assistant to solve the problem without chunking.

basically if you connect to a specific WIFI SID use an INTERNAL_URL otherwise use an EXTERNAL_URL to connect to immich

[Daniel-dev22]

one could implement a feature similar to home assistant to solve the problem without chunking.

basically if you connect to a specific WIFI SID use an INTERNAL_URL otherwise use an EXTERNAL_URL to connect to immich

This isn't really necessary if you use the same hostname for both with DNS. When you're local it points to the local IP and when you're not it points to whatever external IP it is.