MyDLP Blog

If your users generally upload file with larger sizes then usual, you may need to modify Squid timeout options. Because, when MyDLP is inspecting big files (generally several times compressed files ), total inspection time increases naturally. And as a result, Squid could give timeout errors instead of waiting MyDLP.

To make Squid wait MyDLP, in /etc/squid/squid3.conf you should modify these parameters as;

read_timeout 30 minutes
request_timeout 30 minutes
persistent_request_timeout 30 minutes
pconn_timeout 30 minutes

Today, these worked great for us.

Yesterday, in our Squid cache optimization trials “aufs” created problems. We have seen warning lines about “queue congestion in disk I/O operations” in cache.log and at the same time we have detected unreturned HTTP requests.

Then, we have tried “diskd” engine for cache storage system. Results were wonderful. After switching to “diskd”,  warning lines or unreturned requests were disappeared.

Difference between “aufs” and “diskd”, in order to avoid blocking Squid main process with disk I/O requests, “aufs” delegates these requests to new POSIX threads, “diskd” delegates them to another separate process. And in our site, “diskd” gives better results.

I guess we’ll continue with “diskd”.

Several documents about how to optimize Squid cache are available all around the web. Obviously, there is no silver bullet, there is no definite solution.

But these days, we are trying to optimize a Squid cache which is using MyDLP as ICAP server. In our site there are about 1,5K users which are very active web users. Access times are very important and delays create problems.

Our servers have 8-core Xeon CPUs and 16GB of memory.

To improve performance and decrease access times, I have searched through the web and created myself a startup recipe. I’ll also write about results and further changes in this recipe.

Linux commands for setting up OS limits;

echo 1024 32768 > /proc/sys/net/ipv4/ip_local_port_range
echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
ulimit -HSd unlimited
ulimit -HSn 16384

Parameter changes in /etc/squid3/squid.conf file;

icp_port 0
htcp_port 0
icp_access deny all
htcp_access deny all
snmp_port 0
snmp_access deny all
dns_nameservers 127.0.0.1
cache_mem 2048 MB
pipeline_prefetch on
memory_pools on
memory_pools_limit 2048 MB
maximum_object_size 2048 KB
maximum_object_size_in_memory 1024 KB
ipcache_size 4096
ipcache_low 90
ipcache_high 95
cache_dir diskd /var/spool/squid3 4096 16 256
cache_store_log none
log_fqdn off
half_closed_clients off
cache_swap_high 100%
cache_swap_low 80%

Few hours later from now, I’ll push this configuration to one of our servers in production.