Save image based on path to image

[dyanakiev]

Hello, is it possible to save the images based on the requested path to them?
My cache folder started to grow really big.

Example:
https://example.site/img.php?src=uploads/avatars/234234234/avatar22.png
Cache path should be:
/images/cache/uploads/avatars/234234234/avatar22-cached-picture-something-234234.png

[mosbth]

It is related to the cache dir growing?

A crontab script monitoring and removing old files is one way to deal with it.

Option.
One can (pre)process and save images as an "alias" image and use it instead of direct access to img.php.
https://cimage.se/doc/config-file#alias

Could you elaborate a bit more on your perseived problem?

[dyanakiev]

Yes so i dont have problem with the disk space, but the folder itself has really big amount of files and i cant even "ls" in it. I dont need to delete old files but i would like to be able to put the cached image in the cache folder with the same structure as from the url i will try to explain again.

Cache folder: /images/resized_cache/

Request uri: https://example.site/img.php?src=uploads/avatars/234234234/avatar22.png

Now is saved like this:
/images/resized_cache/avatar22-something-2342343.png

I would like to be like this:
/images/resized_cache/uploads/avatars/234234234/avatar22-something-2342343.png

[mosbth]

Ok, so you would like to configure how the structure is created in the cache dir?
Instead of saving all files under cache/ you would prefer it used a subdirectory structure, the same way your original files are stored?
And this could/should be an option one could configure in the configfile?

That could be done. I had some thoughts on this earlier but decided to stick with a flat file structure (easier to implement).

From an interest, what would you gain from having this structure? Nice to have or real benefit?

[flobox]

Big +1!

We also have a huge cache directory (amount of files). On some servers this (huge amount of files in one single directory) could slow down things and cause problems.

[mosbth]

Will it be enough to use a directory structure, in the long term, or should one look into a solution where many images goes into one file (and one index-file)?
Like this:
https://code.facebook.com/posts/685565858139515/needle-in-a-haystack-efficient-storage-of-billions-of-photos/
Perhaps both and start with directory structure and se how long it will be enough.

One could also consider using SQLite for smaller images:
https://www.sqlite.org/fasterthanfs.html

Still, I would like a straight forward solution, without to much hassle.

[flobox]

I think a directory structure would be the most pragmatic solution in my opinion. For my needs the directory structure would solve it!

[dyanakiev]

I think that the directory structure will work best.

[flobox]

Any progress so far regarding this issue? :)

My cache directory is holding 900k of files so far. I have a cronjob enabled deleting cache files older than 365 days, but the directory is still growing. Splitting the cache directory down - like mentioned above - would be super helpful!

[mosbth]

Not much progress, no. I checked my own cache dir and the largest only contains 5k files, not much in comparison to 900k. Nice to know we have some heavy users/usage out there.

I refreshed my memory and gave it some new thougth though. I have no definitive answer for now. I need to sleep on it.

[Surf-N-Code]

@mosbth We are also very much in need of such a feature as well since our cache directory contains more than a million files by now.

So, huge +1 from us as well.

Do you have any plans on integrating the suggested feature? :)

[mosbth]

Plans exists yes. Then, its that thing, with time... and being able to prioritize among all other stuff one has on his magic agenda to conquer the world.

The plan it to build an alternative cache-structure that mirrors the directory structure of the img/ folder. This alternative structure can be turned on in the config file, its default off (to start with).

The directory structure will look something like this:

Source image:

img/image.png

Cache structure:

cache/image.png/h700-w300-cf.png
cache/image.png/h700-w300-cf-a=0,0,50,0.png

So, each image will have its own cache directory where all its cached versions are saved.

I am not sure how well this scales with a million images, but it might make it a tad easier to keep track on files in the cache directory and to get a visual overview of its content.

This should also make it possible to find stray images in the cache, that is images removed/moved in the img/ folder but still remains in the cache/ folder.

There is also the existing cache/fasttrack, where a hit goes straight to the cached image, that should, most likely, go into the new directory structure.

For those having a cache with a million files, one could guess that some kind of transfer process is needed, from old cache to new cache.

So that is the current, waiting to be implemented.

For stats, I looked in my largest website using cimage.

$ du -sk htdocs/img
442296  htdocs/img
$ du -sk cache/cimage/
586516  cache/cimage/
$ ls -R1 htdocs/img/ | wc -l
3059
$ ls -1 cache/cimage/ | wc -l
6237
$ ls -1 cache/cimage/fasttrack/ | wc -l 
7050

As you see, I have a pretty small cache when comparing to a million cached files.

Anyway, I guess I should be real happy to see that some of you are using cimage to an extent of sites with a million cached files - that is really nice to know. Real nice. :-)

[flobox]

@mosbth ah great, see things issue getting picked up!

Right now we are having almost 2 (!) million files in the cache directory. It is starting to get nasty :)
I don't think that your pointed out solution is the greatest for cimage applications with a lot if single files (as it is in our case) - since it would also create a LOT of subfolders in the /cache/ directory. I would rather prefer a "time based" directory structure in the cache directory.

cache/2018/11/
cache/2019/01/
cache/2019/02/

Again: we are super happy with cimage serving so many image files to your users! Thank you for your great work here @mosbth !

[mosbth]

@flobox @Surf-N-Code A quick question, when you say you have millions files in the cache, how many files do you have in the source img/?

@flobox Yes, a time based structure could be an alternative. I would prefer avoiding the need of creating too many sub directories.