Category Archives: Linux/UNIX

Error when trying to FTP !

When trying to FTP-in, facing this error ? :

=================

Status: Resolving address of xxxxxxxxxxxxxxx.com
Status: Connecting to xx.xx.xx.xx:21…
Status: Connection established, waiting for welcome message…
Response: 421 Too many connections (x) from this IP
ons (x) from this IP

=================

As the logs indicate, the limit for connections from the IP you are trying
to login has reached its maximum value.

Increase this from the configuration file, the value 'MaxClientsPerIP'
( if its pure-ftp ) or 'MaxClientsPerHost' ( pro-ftpd)
and restart the service.

Alternatively, you can also terminate the existing connections, if they are not in use.
# netstat -plan | grep :21 and kill the corresponding process
( # kill -9 PID )

Apache error — No space left on device: Couldn’t create accept lock

Apache: [emerg] (28)No space left on device: Couldn’t create accept lock

[notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[crit] (28)No space left on device: mod_rewrite: Parent could not create RewriteLock file /usr/local/apache/logs/rewrite_lock

semget: [emerg] (28) No space left on device OR Apache: No space left on device: Couldn’t create accept lock

You might check if disk space is full and can easily confirm that is not the reason for this error.

The reason behind the error message is Semaphores. You will have to kill the hung/stuck semaphore processes in order

To list the PIDs of the active semaphore processes, execute:

# ipcs -s
—— Semaphore Arrays ——– key
semid owner perms nsems
0×00000000 366673220 apache 600 1
0×00000000 366706589 apache 600 1
0×00000000 366732358 apache 600 1
0×00000000 366734353 apache 600 1

To kill those process, use the command :

# ipcrm -s PID

Once those stuck/hung processes are cleared, restart your apache service.

 

A cPanel bug ( for version — 11.40 ) with clamAV

Getting the following error message ?

===========

Original Message --------
Subject: Cron /usr/local/cpanel/3rdparty/bin/freshclam --quiet --no-warnings
From: (Cron Daemon)
To: root@hostname
Date: 12/12/2013 04:38
> ERROR: Can't create temporary directory

/usr/local/cpanel/3rdparty/share/clamav/clamav-xxxxx.tmp

===========

This is a known issue/bug with cPanel in 11.40

Although the directory ‘/usr/local/cpanel/3rdparty/share/clamav’
has enough permission and ownership configured, it is not able to
create the required files/folders.

A temporary workaround to this issue is to change the ownership of
the directory as shown below :

==========

chown clamav:clamav /usr/local/cpanel/3rdparty/share/clamav

==========

Open resolvers !!

Open resolver ??

Before getting to know what is an open resolver, you need to know what
is recursion, ie recursive queries !

Suppose you have a DNS server configured and a local machine which uses
your DNS server queries for a website. Imagine this query is a new one 
and its not in the local cache of the machine which made the request.
Once this request reaches your DNS server, it will attempt to find the
website in question in its local cache. If it cannot find an answer it
will query other DNS servers on your behalf until it finds the address.
It will then respond to the original request with the results from each
server’s query.

This scenario is fine, because the local machine which made the initial
request is trusted by you.

What if another machine which isn’t trusted by you, queries your DNS server
for the same ? Then your DNS is an Open resolver.

An open DNS resolver is a name server that provides a recursive name resolution
for non local clients or users. Basically it’s a name server that provides recursive
replies for every system on the internet. Local users or “authorized” clients are
users on networks that you control and/or that you trust. Recursive replies are
the result of following the chain of delegations found in DNS, ending up at the
domain name that was requested by the original user.

Open DNS resolvers are frequently being abused to conduct efficient DDoS attacks towards
websites, infrastructure and services. In a DNS amplification DDoS attack, the attacker
sends a DNS name lookup request to an open DNS resolver with the source address
spoofed to be the victim’s address.

When the DNS server sends the DNS record response, it is sent
to the victim (the source address that was used in the spoofed request). Because the size
of the response is typically considerably larger than the request, the attacker is able to
amplify the volume of traffic directed at the victim. Dont think it would affect just the
victim. Essentially this means that your equipment is taking part in a botnet leveraging
a DDoS attack towards other systems, potentially causing disruption of services and harm.

If your systems take part in such a DDoS attack then your own network, server and services
infrastructure too can easily become congested.

To get around this issue, configure your DNS server to either disable recursion or
allow recursion from trusted set of IPs.

recursion can be disabled by adding the following line to your /etc/named.conf file :

options {

recursion no;

};

You can allow recursion from a trusted set of IPs by giving the following

options {

allow-recursion { 127.0.0.1; IP1; IP2; }; //include your server IPs and
your provider’s nameserver IPs and whichever IPs you feel can be trusted
.
};

Suppose you have a DNS server and you have configured your named as

allow-recursion { IP1;IP2; } ;

Try the following from the machine with IP1,

#nslookup google.com x.x.x.x ( x.x.x.x is the DNS server IP )

The result would be :

———–

(root) nameserver = J.ROOT-SERVERS.NET
(root) nameserver = K.ROOT-SERVERS.NET
(root) nameserver = L.ROOT-SERVERS.NET
(root) nameserver = M.ROOT-SERVERS.NET
(root) nameserver = A.ROOT-SERVERS.NET
(root) nameserver = B.ROOT-SERVERS.NET
(root) nameserver = C.ROOT-SERVERS.NET
(root) nameserver = D.ROOT-SERVERS.NET
(root) nameserver = E.ROOT-SERVERS.NET
(root) nameserver = F.ROOT-SERVERS.NET
(root) nameserver = G.ROOT-SERVERS.NET
(root) nameserver = H.ROOT-SERVERS.NET
(root) nameserver = I.ROOT-SERVERS.NET

…..

———–

Suppose you made the same query from an IP which is not
defined in allow-recursion, then you get the following

———-

Server: x.x.x.x
Address: x.x.x.x#53

** server can’t find google.com: REFUSED

———-

So, consider about tweaking your DNS server, if its an Open resolver !

csf & iptables cheatsheet !!

CSF

csf -a   : allow an ip and add it to /etc/csf.allow
csf -ar  : remove an ip from /etc/csf.allow and delete rule
csf -d   : deny an ip and add to /etc/csf.deny
csf -dr  : unblock an ip and remove from /etc/csf.deny
csf -g   : search the list and give the rule that matches the ip
csf -tr : Remove the IP from temporary ban
csf -x   : disable csf and lfd
csf -e   : enable csf and lfd if disabled
csf -r   : restart csf

CSF config files

  • /etc/csf/csf.conf     :csf config file
  • /etc/csf/csf.allow    :csf allow file
  • /etc/csf/csf.deny     :csf deny file
  • /etc/csf/csf.ignore   :ignore list file ( the ip’s lfd should ignore and not block )
  • /etc/csf/csf.tempban  :to see the ips in temporary ban

To block an entire range of IP’s from a country

Open CSF config file and check for the line  “CC_DENY”  and add the corresponding country code.

For eg, if you want to block the IPs from china, add the country code as ‘CN’

IPTABLES

service iptables status : display the status of firewall
iptables -F :flush out rules
iptables -L -INPUT -n : check the lines of the chain input
iptables -I INPUT -s x.x.x.x -j DROP   : block a single ip address
iptables -D INPUT -s x.x.x.x -j DROP   : delete the ip from the rule
iptables -A INPUT -s x.x.x.x -j ACCEPT : allow all traffic from the ip address
iptables -A INPUT -p tcp --dport 3306 -j DROP : block a port from all ip
iptables -A INPUT -p tcp -s x.x.x.x --dport 3306 -j ACCEPT : allow a port from a single ip
iptables -A INPUT -m mac --mac-source 00:0F:EA:91:04:08 -j DROP :
block traffic from mac address

Diff b/w DROP and REJECT : REJECT works like DROP, but will return an error
message to the host sending the packet that the packet was blocked

iptables-save > /root/rule.file: To save iptables rules to an external file
iptables-restore < /root/rule.file
: To restore the rules back

iptables -L INPUT --line-numbers : To list the rules along with the rule
number in the chain 'INPUT'
iptables -D INPUT 1 : To delete the rule 1 in the chain INPUT

Options +Includes — what is it ?

What is the option seen as Options +Includes ??

SSI (Server Side Includes) are directives that are placed in HTML pages, and evaluated on the
server while the pages are being served. They let you add dynamically generated content to an existing HTML page, without having to serve the entire page via a CGI program, or other dynamic technology.

The decision of when to use SSI, and when to have your page entirely generated by some program, is usually a matter of how much of the page is static, and how much needs to be recalculated every time the page is served.

SSI is a great way to add small pieces of information, such as the current time.

To permit SSI on your server, you must have the following directive either in your httpd.conf file, or in a .htaccess file:

Options +Includes

Issue with Apache and SymLinks

The vulnerability with Symlinks and Apache is a known issue
in a shared hosting environment.

1st step employed by the attacker in order to carry out this attack it to find a compromised ‘single’ website or domain which has got any vulnerable scripts or 3rd party applications or any themes used in it. Once he get access to a single domain, he moves forward by creating the symlinks to other websites or even he can symlink to / (root).

For eg, if you have the following symlink set in any domain,

link -> /root , using the directory ‘link’ anyone can actually access
/root and can access any sensitive file.

Rather than manually creating this sort of symlinks, the hacker can even use any
perl/cgi script to create a symlink to other users of the server.

As a basic soultion for this, you can ensure that Apache is configured in a
way so as not to following symlinks (Options -FollowSymLinks)

================

To disable the ability for Apache to allow users to follow symbolic links in their requests, remove the FollowSymLinks directive on your Directory commands.

For example, if the below was the configuration then,

<Directory "/usr/local/apache/htdocs">
Options Indexes FollowSymLinks
AllowOverrride None
Order allow,deny
Allow from all

Remove the FollowSymLinks reference so that this reads:

<Directory "/usr/local/apache/htdocs">
Options Indexes
AllowOverrride None
Order allow,deny
Allow from all

================

If you really need symlinks, you can use the “SymLinksIfOwnerMatch” option to only
allow links from within the same user.

To prevent PHP from accessing any file outside of their directory, you need to specify the ‘open_basedir’ setting ( in PHP configuration file ) to only have access to their directory.

This option can be enabled from WHM, but :

==========

This security tweak uses Apache DSO style directives. If PHP is
configured to run as a CGI, SuPHP or
FastCGI process, the open_basedir setting must be manually specified
in the relevant php.ini file.
See the EasyApache documentation for more information.

==========

If the PHP handler is set as CGI or SuPHP, then tweak settings seen in WHM
cannot be used to set the openbase_dir option.

You need to manually specify the openbase_dir option in the global
PHP configuration file ( use php -i |grep php.ini to find the php.ini location )

In addition to prevent this SymLinks attack, there are various patches too :

https://forums.cpanel.net/f185/solutions-handling-symlink-attacks-202242-p4.html#post996441

To be kept in mind is :: the root cause for this attack or vulernablity is due any
unsecured scripts/plugins/applications which might be employed in any of the domains.

ip_conntrack: table full, dropping packet !!

Facing an issue with the kernel module, ‘ip_conntrack’ ?

Checking /var/log/messages gives something like this ?

==========

Nov 13 14:45:23 host kernel: ip_conntrack: VPS xxx table full, dropping packet.
Nov 13 14:45:43 host kernel: ip_conntrack: VPS xxx table full, dropping packet.
Nov 13 14:45:48 host kernel: ip_conntrack: VPS xxx table full, dropping packet.

==========

If you run an iptables firewall, and have rules that act upon the state of a packet,
then the kernel uses ip_conntrack to keep track of what state what connections are in so that
the firewall rule logic can be applied against them. If you have a system that’s getting a lot
of network activity then the table will accumulate entries.

* Increase ip_conntrack to a higher value by editing /etc/sysctl.conf

Add/edit this line,

net.ipv4.ip_conntrack_max=xxxx

Run , #sysctl -p after making the changes.

Check the current value using the command,

# sysctl net.ipv4.netfilter.ip_conntrack_max

Dont keep on increasing the above value (ip_conntrack_max) beyond a limit, if you still see the error after the increase. This error might indicate the start of something more destructive attack on your servers network, something like a DDoS attacks. The amount of packets sent/received during this period would be on the higher side and as a result the kernel module isnt able to process them all, which results in the above error.

So check for the server traffic using commands like iftop or tcpdump and isolate if the
issue is related to any attacks.

Dovecot issue – dovecot.index file broken ?

Dovecot issue – dovecot.index file corrupted?

Any email user not able to access via his webmail? Does it show
logins failed, even if you are cent percent sure logins are correct?

Check /var/log/maillog.

# tailf /var/log/maillog

If you find anything like dis,

=============

Nov 12 20:59:02 host dovecot: imap(zzzz@yyyy.com): Error: Transaction log file /home/xxxx/mail/yyyy.com/zzzz/dovecot.index.log seq 302:

Nov 12 20:59:02 host dovecot: imap(zzzz@yyyy.com): Error: broken sync positions in index file /home/xxxx/mail/yyyy.com/zzzz/dovecot.index

Nov 12 20:59:02 host dovecot: imap(zzzz@yyyy.com): Warning: fscking index file /home/xxxx/mail/yyyy.com/zzzz/dovecot.index

Nov 12 20:59:02 host dovecot: imap(zzzz@yyyy.com) Error: Fixed index file /home/xxxx/mail/yyyy.com/zzzz/dovecot.index log_file_tail_offset 1184 -> 988

Nov 12 20:59:02 host dovecot: imap(zzzz@yyyy.com): Panic: file mail-transaction-log.c: line 350 (mail_transaction_log_set_mailbox_sync_pos): assertion failed: (file_offset >= log->head->saved_tail_offset)

=============

As indicated in the logs, there seems to be an issue with the dovecot index file for the user ‘zzzz’. The basic idea behind Dovecot’s index files is that it makes reading the mailboxes a lot faster.

This happens to be a long term issue with dovecot.

The solution to fix this issue is to delete dovecot.index file.

 

Virtuozzo – Basics

Virtuozzo is a software application for enterprise server virtualization that allows an administrator to create virtual environments on a host computer at the operating system (OS) layer. Instead of having one physical machine run multiple operating systems simultaneously, as the virtual machine model used by VMware, Virtuozzo approaches virtualization by running a single OS kernel as its core and exporting that core functionality to various partitions on the host.

Each of the partitions effectively becomes a stand-alone entity called a virtual private server (VPS)

Installation in a CentOS box:
Before proceeding to the installation of virtuzzo make sure you have the partition /vz or
create it if you are installing on a fresh server

/vz contains all container data and parallels virtuzzo containers templates

INSTALLATION
Download the vzinstall-linux-x86_64.bin utilty from the oficial site.
Make the script executable by # chmod a+x vzinstall-linux-x86_64.bin
Run the script by # ./vzinstall-linux-x86_64.bin

You will get the following wizard :
Either you can download and install or install for future or on any other computer.
The configure options allow you to configure the various parameters that the virtuozzo
containers use during the execution. If you select the option download only, after the download is over, go the download directory (root/virtuzzo/Download ) and copy the content of this directory to the system where you are planning to install virtuzzo and execute the following script:

# ./virtuozzo-4.7.0-<build_version>-x86_64.sfx

If you select the option download and install you can either do it in 3 ways:
Default: Select this radio button to download and install the Parallels Virtuozzo Containers program files and one OS template—CentOS 5 (you will need this OS template to create Containers on its basis).

Full: Select this radio button to download all available OS templates to the server and install them there.

Custom: Select this radio button to customize the set of OS templates to download to and install on the server. In this case, once you click the Next button, you will see the Select Templates window where you can choose the necessary OS templates for downloading

In the next step of wizard, click download to start download paralells virtuzzo containers and selected templates to the server.

In the next step you would be asked for the license key.

Install a valid Parallels license by entering the license key number in the field provided and clicking Next. If you plan to activate Parallels Virtuozzo Containers with an activation code,make sure that your server is connected to the Internet

Finally, the installation program displays the Congratulations window.

Leave the Install PVA Agent and Install PVA Management Node check boxes selected to set up the Parallels Virtual Automation application and its components on the server once you restart it. With Parallels Virtual Automation, you can connect to the server and manage Containers using your favorite browser. If you select both check boxes, the installer does the following after restarting the server:

1. Downloads the installation packages for Parallels Virtual Automation from the Parallels website.

2. Install the PVA Agent component on the server. PVA Agent ensures the interaction between your server, the Management Node (see below), and Parallels Virtual Automation. Without this component installed, you will not be able to connect to your server using Parallels Virtual Automation.

3. Creates a special Container on the server and installs the PVA Management Node
component inside it. PVA Management Node (also called Master Server) ensures the
communication between the server running Parallels Virtuozzo Containers (known as Slave
Server) and the Parallels Virtual Automation application. The Master Server keeps a
database with the information about all registered Slave Servers.

If you have already set up a Master Server, you can skip this step (clear Install PVA Management Node check box).

After this step you will be asked for the IP address and hostname and DNS of the container which

will act as the PVA management node.

To log in to Parallels Virtual Automation, launch a Web browser compatible with PVA

The list of currently supported Web browsers is given below:

• Internet Explorer 6.0 and above
• Firefox 2.x and above
• Safari 3.x and above

On the Master Server or any other computer, open your favorite Web browser and log in to Parallels Virtual Automation by typing the Master Server IP address or hostname and TCP port 4648 in the address bar.

http://ipaddressofpvm:4648
Login using the username and password of the container which acts as the PVM

Manually setting up PVA and management node
Create the container : vzctl create CTID –ostemplate centos-6-x86_64 –hostname “hostname”
Set the ip address and nameserver for the created container which will act as the MN

# vzctl start CTID
# vzpkg install CTID -p perl-DBI

Download PVA Management Node installer

# wget http://download.pa.parallels.com/pva/pv ... loy.x86_64
# chmod a+x pva-setup-deploy.x86_64
# ./pva-setup-deploy.x86_64 -d /vz/root/CTID/root/ --extract
# vzctl enter CTID
# cd /root
# ./pva-setup --install

.htaccess files – Basics

A small note on .htaccess file.

What is .htaccess?

.htaccess is a configuration file for use on web servers running the Apache Web Server. When an .htaccess file is placed in a directory it is detected by the web server and gets exectued.These .htaccess files are used to alter the configuration of the Apache Web Server to enable/disable additional functionality and features that the Apache Web Server software has to offer.

A sample .htaccess file :

AuthName "security check"
AuthUserFile /path/to/password/file/.htpasswd
AuthType Basic
require valid-user
ErrorDocument 401 /error_pages/401.html

As per the above .htaccess file, it enables password protection on the directory; it offers redirection to a custom error page if a user fails to login correctly.

This is just a basic example.

.htaccess files are very powerful and they can be extremely fine tuned to meet your needs.

Protect your server from DDoS attacks – Part 1 !!

What is a DDoS attack ?

DDoS, short for Distributed Denial of Service, is a type of DOS attack where multiple compromised systems — which are usually infected with a Trojan (70% of the time)– are used to target a single system causing a Denial of Service (DoS) attack. Victims of a DDoS attack consist of both the end targeted system and all systems maliciously used and controlled by the hacker in the distributed attack.

In a DDoS attack, the incoming traffic flooding the victim originates from many different sources – potentially hundreds of thousands or more. This effectively makes it impossible to stop the attack simply by blocking a single IP address.

Honestly, it would be so difficult to protect against a DDoS attack. But we can follow some steps to make our servers more watchful against them.

=====================================

CSF ( WHM default firewall ) can be fine tuned as follows :

ConfigServer Security & Firewall from WHM >> Firewall Configuration

Connection Tracking : This option enables tracking of all connections from IP
addresses to the server. If the total number of connections is greater than
this value then the offending IP address is blocked. This can be used to help
prevent some types of DOS attack.

Care should be taken with this option. It’s entirely possible that you will
see false-positives. Some protocols can be connection hungry, e.g. FTP, IMAPD
and HTTP so it could be quite easy to trigger, especially with a lot of
closed connections in TIME_WAIT. However, for a server that is prone to DOS
attacks this may be very useful. A reasonable setting for this option might
be around 300.

=====================================
If you see your server is a bit on the slower side, check the number of connections to it using the following command.

# netstat -anp |grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

As a preliminary step, block the IPs which doesn look valid and are offending ones using csf commands.

Another option is to go for the the MOD_EVASIVE module in the httpd configuration.

Mod_evasive is an evasive maneuvers module for Apache to provide evasive
action in the event of an HTTP DoS or DDoS attack or brute force attack.
It is also designed to be a detection tool and can be easily configured to talk
to ipchains, firewalls.

Mod_evasive have got many many options to gun down our requirements to handle the
IPs connecting to our server.

Steps to install mod_evasive is given below :

# cd /usr/local/src/

# wget http://www.zdziarski.com/blog/ wpcontent/uploads/2010/02/mod_evasive_1.10.1.tar.gz

# tar -xvzf mod_evasive_1.10.1.tar.gz

# cd mod_evasive/

# /usr/local/apache/bin/apxs -cia mod_evasive20.c

Now create a file named /usr/local/apache/conf/mod_evasive.conf and add your custom settings.

For eg :

# cat /usr/local/apache/conf/mod_evasive.conf

LoadModule evasive20_module modules/mod_evasive20.so
<IfModule mod_evasive20.c>

DOSHashTableSize 3097

((The hash table size defines the number of top-level nodes for each child’s hash table. Increasing this number will provide faster performance by decreasing the number of iterations required to get to the record, but consume more memory for table space. You should increase this if you have a busy web server. The value you specify will automatically be tiered up to the next prime number in the primes list))

DOSPageCount 2

((This is the threshhold for the number of requests for the same page (or URI) per page interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list))

DOSSiteCount 50

((This is the threshhold for the total number of requests for any object by the same client on the same listener per site interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list))

DOSPageInterval 1

((The interval for the page count threshhold; defaults to 1 second intervals))

DOSSiteInterval 1

((The interval for the site count threshhold; defaults to 1 second intervals))

DOSBlockingPeriod 10

((The blocking period is the amount of time (in seconds) that a client will be blocked for if they are added to the blocking list. During this time, all subsequent requests from the client will result in a 403 (Forbidden) and the timer being reset (e.g. another 10 seconds). Since the timer is reset for every subsequent request, it is not necessary to have a long blocking period; in the event of a DoS attack, this timer will keep getting reset))

</IfModule>

Now include the above file inside /usr/local/apache/conf/includes/pre_main_global.conf

Include “/usr/local/apache/conf/mod_evasive.conf

Now rebuild httpd.conf
# /scripts/rebuildhttpdconf

Now restart apache
# /scripts/restartsrv httpd

Steps in hardening your mail service :

Some of the things which can be done from WHM to harden your mail service :

=========================
From WHM Main >> Server Configuration >> Tweak Settings:
* POP3 connection limit option prevents lots of POP3 connections.
* POP3 flood prevention option.
* Prevent “nobody” from sending mail : This will ensure that PHP
scripts user the ownership of user “nobody” will not be able send any mails.
* In service manager you can find the option “antirelay” . Turn
this off so that each time POP3 connects authentication would be required.

=========================

Try to use Secure protocols and related ports

POP3S 995
IMAPS 993
SMTPS 465

These are just basics in hardening the mail system. More ones to follow……