MyDLP Blog

Yesterday, in our Squid cache optimization trials “aufs” created problems. We have seen warning lines about “queue congestion in disk I/O operations” in cache.log and at the same time we have detected unreturned HTTP requests.

Then, we have tried “diskd” engine for cache storage system. Results were wonderful. After switching to “diskd”,  warning lines or unreturned requests were disappeared.

Difference between “aufs” and “diskd”, in order to avoid blocking Squid main process with disk I/O requests, “aufs” delegates these requests to new POSIX threads, “diskd” delegates them to another separate process. And in our site, “diskd” gives better results.

I guess we’ll continue with “diskd”.

Several documents about how to optimize Squid cache are available all around the web. Obviously, there is no silver bullet, there is no definite solution.

But these days, we are trying to optimize a Squid cache which is using MyDLP as ICAP server. In our site there are about 1,5K users which are very active web users. Access times are very important and delays create problems.

Our servers have 8-core Xeon CPUs and 16GB of memory.

To improve performance and decrease access times, I have searched through the web and created myself a startup recipe. I’ll also write about results and further changes in this recipe.

Linux commands for setting up OS limits;

echo 1024 32768 > /proc/sys/net/ipv4/ip_local_port_range
echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
ulimit -HSd unlimited
ulimit -HSn 16384

Parameter changes in /etc/squid3/squid.conf file;

icp_port 0
htcp_port 0
icp_access deny all
htcp_access deny all
snmp_port 0
snmp_access deny all
dns_nameservers 127.0.0.1
cache_mem 2048 MB
pipeline_prefetch on
memory_pools on
memory_pools_limit 2048 MB
maximum_object_size 2048 KB
maximum_object_size_in_memory 1024 KB
ipcache_size 4096
ipcache_low 90
ipcache_high 95
cache_dir diskd /var/spool/squid3 4096 16 256
cache_store_log none
log_fqdn off
half_closed_clients off
cache_swap_high 100%
cache_swap_low 80%

Few hours later from now, I’ll push this configuration to one of our servers in production.

Mail, printer and removable media traffic archives;

We are planning to add MyDLP some archiving capabilities. After a few weeks, MyDLP will be able to archive every or selected data flows through mail, printer channels and removable media. System administrators will be able to trace every single data traffic.

Also, very fascinating performance improvements are upcoming…

Stay tuned…

MyDLP was supporting only two compression algorithms; ZIP and RAR. ZIP has been supported with usage of OTP’s ZIP library. But, to unRAR a compressed file, we were executing the “unrar” binary.

After several talks with users, it became clearer  that our support for compressed file formats was a little bit poor. And we decided that we need more.

With integration of 7z, several compression formats started to be supported by MyDLP.  ZIP, RAR, CAB, ARJ, GZIP, BZIP2, TAR, CPIO, RPM and DEB; now contents for all of them could be inspected by MyDLP.

We have added two new documents  to the MyDLP Documents library.

• For MyDLP Server HLD, click here

• For MyDLP Endpoint Registry Entries, click here

We are planning to allocate some time to improve our documentation. Let us know your document requests.