MyDLP Blog

easy, simple, open source data leakage prevention

Archive for the ‘Squid’ Category

Squid timeouts

without comments

If your users generally upload file with larger sizes then usual, you may need to modify Squid timeout options. Because, when MyDLP is inspecting big files (generally several times compressed files ), total inspection time increases naturally. And as a result, Squid could give timeout errors instead of waiting MyDLP.

To make Squid wait MyDLP, in /etc/squid/squid3.conf you should modify these parameters as;

read_timeout 30 minutes
request_timeout 30 minutes
persistent_request_timeout 30 minutes
pconn_timeout 30 minutes

Today, these worked great for us.

Written by kerem

March 15th, 2011 at 9:39 pm

Posted in Performance,Squid

Diskd seems better than aufs

without comments

Yesterday, in our Squid cache optimization trials “aufs” created problems. We have seen warning lines about “queue congestion in disk I/O operations” in cache.log and at the same time we have detected unreturned HTTP requests.

Then, we have tried “diskd” engine for cache storage system. Results were wonderful. After switching to “diskd”,  warning lines or unreturned requests were disappeared.

Difference between “aufs” and “diskd”, in order to avoid blocking Squid main process with disk I/O requests, “aufs” delegates these requests to new POSIX threads, “diskd” delegates them to another separate process. And in our site, “diskd” gives better results.

I guess we’ll continue with “diskd”.

Written by kerem

February 25th, 2011 at 8:28 am

Posted in Performance,Squid

Squid3 optimization tips

without comments

Several documents about how to optimize Squid cache are available all around the web. Obviously, there is no silver bullet, there is no definite solution.

But these days, we are trying to optimize a Squid cache which is using MyDLP as ICAP server. In our site there are about 1,5K users which are very active web users. Access times are very important and delays create problems.

Our servers have 8-core Xeon CPUs and 16GB of memory.

To improve performance and decrease access times, I have searched through the web and created myself a startup recipe. I’ll also write about results and further changes in this recipe.

Linux commands for setting up OS limits;

echo 1024 32768 > /proc/sys/net/ipv4/ip_local_port_range
echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
ulimit -HSd unlimited
ulimit -HSn 16384

Parameter changes in /etc/squid3/squid.conf file;

icp_port 0
htcp_port 0
icp_access deny all
htcp_access deny all
snmp_port 0
snmp_access deny all
cache_mem 2048 MB
pipeline_prefetch on
memory_pools on
memory_pools_limit 2048 MB
maximum_object_size 2048 KB
maximum_object_size_in_memory 1024 KB
ipcache_size 4096
ipcache_low 90
ipcache_high 95
cache_dir diskd /var/spool/squid3 4096 16 256
cache_store_log none
log_fqdn off
half_closed_clients off
cache_swap_high 100%
cache_swap_low 80%

Few hours later from now, I’ll push this configuration to one of our servers in production.

Written by kerem

February 24th, 2011 at 11:56 am

Posted in Performance,Squid

Now squid3-ssl packages in MyDLP repository

with 2 comments

Everybody who played with Squid on Ubuntu, have probably encountered with this problem; Ubuntu Squid packages had been compiled without SSL option. Therefore, it is not possible to proxy HTTPS connections with Squid on Ubuntu Server.

So, what should we do in order to overcome this?
If I compile Squid from original source tree, I will be forced to follow new releases and security updates for Squid manually. APT will become useless for me.
If I get dpkg source of Ubuntu Squid package with apt-get source squid3, change debian/rules, add --enable-ssl to configure, recompile it and deploy it instead of original Ubuntu Squid package (as explained here), SSL support will dissappear after installing first update for Squid from official Ubuntu repository.
We have solved all these by creating a new package named squid3-ssl. squid3-ssl replaces squid3, conflicts with squid3 and provides squid3 (dpkg terms). In other words, both squid3 and squid3-ssl could not be installed together, you could install squid3-ssl instead of squid3 and this situation will not cause any dependency problems.
Furthermore, these packages will be automatically compiled after any update for Squid in official Ubuntu repositories and deployed into MyDLP Ubuntu repository.
As a result, adding MyDLP repository and installing squid3-ssl package, will provide Squid SSL functionality without challenging any of the problems I have mentioned above.
To use and install squid3-ssl packages;
Add this line to /etc/apt/sources.list ;

deb lucid/

Then, call aptitude to update itself and install squid3-ssl ;
sudo aptitude update
sudo aptitude install squid3-ssl
That’s all…
Have a good day :)

Written by kerem

November 8th, 2010 at 6:50 pm

Posted in HowTo,Squid,Ubuntu

Squid ICAP Configuration Example

with 3 comments

The upcoming MyDLP SaaS will be working with your preferred ICAP enabled proxy server. So we are going to post ICAP configuration examples for widely used proxies, starting with Squid. Squid is an open source caching proxy which has a very large user base. It is open source and very stable.

Squid 3.0 and 3.1 have out of the box ICAP support with slightly different configuration parameters. Unfortunately, older versions have no built-in ICAP support but it is possible to enable it using patches.

For Squid 3.1 enter following lines into squid.conf file to enable request adaptation:

icap_enable on

icap_service service_req reqmod_precache bypass=0 icap://
adaptation_access service_req allow all

For Squid 3.0 enter following lines into squid.conf file to enable request adaptation:

icap_enable on

icap_service service_req reqmod_precache 0 icap://
icap_class class_req service_req
icap_access class_req allow all

There are more parameters for configuring ICAP which are explained here.

Written by ozgur

October 26th, 2010 at 8:23 am

Posted in ICAP,Squid