Caching large amounts of data
We need to cache all psd/zip files from our amazone S3 storage. The amount of the cache between 150-200 Gb (we need to store files at least 2 month). Now we use transparent proxy squid without any caching.
So my question: is it possible to configure "main" squid such way that all queries to the s3.amazonaws.com he will proxy to the parent squid. In the parent squid we need to cache all zip and psd files only from s3.amazonaws.com. If there are no such files in the cache the parent squid will be download and put "new" files in the cache.
How will squid works with files 1-2 gb? Is there any limitations?
The main proof of this setup is that all psd/zip files from amazone storage delivered to the local clients with maximal speeds from local cache. Because the same files at the same time used by different employees and we spent a lot of time to download those files by each employees
Can this issue be solved in different ways?
We have tested our system about month and it's worked fine. if someone is interested, below I have added configuration files (main/parent squid)
main squid (192.168.100.1) configuration file
acl local_net src 192.168.0.0/16
acl parent_peer peername PARENT_PEER
acl parent_squid dst 192.168.100.50
acl FILE_TO_CACHE urlpath_regex \.(zip|iso|rar)$
acl STORAGE dstdomain storage.example.net
http_port 192.168.100.1:3128 intercept
icp_port 3130
cache_peer 192.168.100.50 parent 3128 3130 name=PARENT_PEER connect-timeout=7 proxy-only
cache_peer_access PARENT_PEER allow STORAGE FILE_TO_CACHE
# to connect to parent via internal interface
tcp_outgoing_address 192.168.100.1 parent_peer
# to properly get cache digest from parent
tcp_outgoing_address 192.168.100.1 parent_squid
# sent all other packets via ISP2
tcp_outgoing_address xxx.xxx.xxx.xxx local_net
parent squid (192.168.100.50) configuration file
acl main_squid src 192.168.100.1
acl FILE_TO_CACHE urlpath_regex -i \.(zip|iso|rar)$
cache allow FILE_TO_CACHE
cache deny all
http_access allow main_squid
http_access allow localhost
http_access deny all
icp_port 3130
icp_access allow main_squid
icp_access deny all
http_port 192.168.100.50:3128
cache_mem 12 GB
maximum_object_size_in_memory 64 MB
minimum_object_size 16384 KB
maximum_object_size 1024 MB
cache_swap_low 93
cache_swap_high 98
refresh_pattern \.iso$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.zip$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.rar$ 129600 100 129600 override-lastmod override-expire ignore-reload
cache_effective_user squid
cache_effective_group squid