Unzip all zip files in the current folder

for f in *.zip;do unzip "$f";done

Distributed key-value stores

http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ - Anti-RDBMS: A list of distributed key-value stores

http://randomfoo.net/2009/04/20/some-notes-on-distributed-key-stores - Some Notes on Distributed Key Stores

http://bret.appspot.com/entry/how-friendfeed-uses-mysql - How FriendFeed uses MySQL to store schema-less data

http://anyall.org/blog/2009/04/performance-comparison-keyvalue-stores-for-language-model-counts/ - Performance comparison: key/value stores for language model counts

http://michalfrackowiak.com/blog:redis-performance - Redis Performance on EC2 (aka weekend project coming)

http://labs.gree.jp/Top/OpenSource/Flare-en.html - Flare is distributed, and persistent key-value storage - pluggable storage (currently only Tokyo Cabinet is available, though:)

http://blip.tv/file/1949416/ - Drop ACID and think about data

http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/ - Tokyo Cabinet: Beyond Key-Value Store

http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation - Tokyo Cabinet and Tokyo Tyrant Presentation

Installing Tokyo Tyrant on Ubuntu 9.04 (Jaunty), in custom folders/directories

Tokyo Cabinet - first install this
-------------
cd /usr/local/src/
wget http://tokyocabinet.sourceforge.net/tokyocabinet-1.4.21.tar.gz
tar xvf tokyocabinet-1.4.21.tar.gz
cd tokyocabinet-1.4.21
apt-get install zlib1g-dev
apt-get install libbz2-dev
mkdir /usr/local/tokyocabinet-1.4.21/
./configure --prefix=/usr/local/tokyocabinet-1.4.21/
make
make install

Tokyo Tyrant
------------
cd /usr/local/src/
wget http://tokyocabinet.sourceforge.net/tyrantpkg/tokyotyrant-1.1.27.tar.gz
tar xvf tokyotyrant-1.1.27.tar.gz
cd tokyotyrant-1.1.27
mkdir /usr/local/tokyotyrant-1.1.27
./configure --prefix=/usr/local/tokyotyrant-1.1.27/ --with-tc=/usr/local/tokyocabinet-1.4.21/
make
make install

Fix lib problem
------------
cd /usr/local/tokyotyrant-1.1.27/bin/
./ttserver

You will get this error: ./ttserver: error while loading shared libraries: libtokyocabinet.so.8: cannot open shared object file: No such file or directory

To fix it,
cd /usr/local/tokyotyrant-1.1.27/
ln -s /usr/local/tokyocabinet-1.4.21/lib/libtokyocabinet.so.8 lib/
./bin/ttserver will now work

Easily changing the timezone in Ubuntu server

dpkg-reconfigure tzdata

How to delete a certain line from a file in Linux

sed -i 321d /path/to/file

where 321 is the line number

Ramdisk in Linux/Ubuntu

sudo mkdir /mnt/ramdisk
sudo mount -t tmpfs tmpfs /mnt/ramdisk -o size=100M,nr_inodes=200k,mode=01777

Pipe Viewer

"Pipe viewer is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion."

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

Protecting from a DDOS SYN FLOOD on port 80 (Apache, Ubuntu)

Other keywords: SYN_RECV, TIME_WAIT

sysctl -w net.ipv4.tcp_synack_retries="1"
sysctl -w net.ipv4.tcp_max_syn_backlog="40000"
sysctl -w net.ipv4.netfilter.ip_conntrack_max="200000"
sysctl -w net.ipv4.tcp_fin_timeout="3"

/sbin/iptables -A INPUT -i eth2 -p tcp --tcp-flags ALL ACK,RST,SYN,FIN -j DROP
/sbin/iptables -A INPUT -i eth2 -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
/sbin/iptables -A INPUT -i eth2 -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
(you can ignore the iptables part, probably doesn't help that much)

In Apache, increase the maximum number of concurrent connections.

I also learned that while in theory putting a squid in reverse proxy on port 80 to protect the Apache behind it is a good idea, squid sucks at it.

Oh, I almost forgot. Make sure syn_cookies is active (if you kernel supports it):
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Google is your friend for further details on the above settings.

More:
http://tools.ietf.org/html/rfc4987#section-3.5
http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_9-4/syn_flooding_attacks.html

Merge/Combine PDF files in Ubuntu/Linux

apt-get install pdftk

then read this http://www.accesspdf.com/article.php/20041129175231241

Linux/Ubuntu Shift keyboard bug when using VMware Player or Workstation

Symptom: "your keyboard outside the vm client stops working correctly no capslock, shift nor control is working" - https://bugs.launchpad.net/ubuntu/+bug/187165 Same bug here also: https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/195982

Fix: Run the command setxkbmap in a terminal window

Update: Seems to be fixed in VMware Workstation 6?

2008 GSOC Participants

View Larger Map

Connecting Ubuntu Linux 8.04 (Hardy) to eduroam (INHOLLAND Diemen)

1) Make sure you are root (sudo su)

2) Create a file called /etc/wpa_supplicant/wpa_supplicant.conf and add the following lines in it:

network={
ssid="eduroam"
scan_ssid=1
key_mgmt=WPA-EAP
eap=PEAP
identity="STUDENT_ID@student.inholland.nl"
password="STUDENT_PASSWORD"
phase1="peaplabel=0"
phase2="auth=MSCHAPV2"
subject_match="CN=radius.inholland.nl"
}

Replace STUDENT_ID with your student number and STUDENT_PASSWORD with our password. If you copy/paste from this document, make sure that the quotes (“) above are real quotes, and not a character that looks like a quote.

3) Run the following command (as root):

sudo wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant.conf -iwlan0 -Dwext

where wlan0 is your wireless interface. Don't close the terminal.

4) Run dhclient wlan0 as root in another terminal, where wlan0 is your wireless interface:

sudo dhclient wlan0

After you get an IP address you can close this terminal, but leave the other one open.

Happy browsing!

Troubleshooting:
If you get certificate error problems, make sure that the contents of /etc/ssl/certs/ca.pem is the same as the file located here: http://secure.globalsign.net/cacert/sureserverEDU.pem

Too many open files - change ulimit values in Ubuntu 8.04 (Hardy Heron)

1) Add line

* hard nofile 51200

to /etc/security/limits.conf

Note: The * means every user except root
Note: 51200 represents the number of concurrent open files. This number must by a multiple of 1024

2) Add line

session required pam_limits.so

to /etc/pam.d/common-session

3) Stop all processes that need the new limit

4) Restart SSH server

5) Login again

6) Run ulimit -a to check the values

7) If the values haven't changed, reboot and try step 6 again

8) Start new processes from command line

Upgrade Ubuntu 6.10 (Edgy Eft) to 7.04 (Feisty Fawn) to 7.10 (Gutsy Gibbon), then to 8.04 (Hardy Heron)

From 6.10 to 7.04:

sed -e 's/\edgy/feisty/g' -i /etc/apt/sources.list
apt-get update && apt-get upgrade && apt-get dist-upgrade

From 7.04 to 7.10:

apt-get install update-manager-core
do-release-upgrade

From 7.10 to 8.04.1:

do-release-upgrade

Remove/delete/clean .svn folders

find /absolute/path/to/folder -name ".svn" -exec rm -rf {} \;

VPN Server + NAT on Ubuntu

(should work with Microsoft Windows Vista/XP and Apple Mac OS Panther/Tiger/Leopard etc clients also)

Tested on Ubuntu Hardy (8.04)

1) Install pptpd

sudo apt-get install pptpd

2) Edit /etc/pptpd.conf and add the following two lines:

localip 10.2.2.1
remoteip 10.2.2.2-5

localip is the address of t he VPN (ppp0) interface on the VPN server (this interface will be create automatically).
remoteip is the range of IP addresses that will be given to the VPN clients

Please add an empty newline at the end of /etc/pptpd.conf, or the daemon will hang.

3) Edit /etc/ppp/chap-secrets and add one line for each VPN user:

USERNAME pptpd PASSWORD *

where USERNAME is the username, pptpd should remain as it is, and PASSWORD is the password. The * at the end means that this particular user can connect to the VPN server from any IP address. You couldd replace it with an ip address range:

"Any following words on the same line are taken to be a list of acceptable IP addresses for that client. If there are only 3 words on the line, or if the first word is "-", then all IP addresses are disallowed. To allow any address, use "*". A word starting with "!" indicates that the specified address is not acceptable. An address may be followed by "/" and a number n, to indicate a whole subnet, i.e. all addresses which have the same value in the most significant n bits. In this form, the address may be followed by a plus sign ("+") to indicate that one address from the subnet is authorized, based on the ppp network interface unit number in use. In this case, the host part of the address will be set to the unit number plus one." (pptpd manual)

4) Add the following lines in /etc/rc.local

# pptp VPN NAT
iptables -t nat -A POSTROUTING -s 10.2.2.0/24 -o eth0 -j MASQUERADE
iptables -A FORWARD -s 10.2.2.0/24 -o eth0 -j ACCEPT
iptables -A FORWARD -d 10.2.2.0/24 -m state --state ESTABLISHED,RELATED -i eth0 -j ACCEPT
route add -net 10.2.2.0 netmask 255.255.255.0 dev ppp0

where eth0 is your "Internet" interface on the server, the one connected directly to the Internet.

5) Start pptpd

/etc/init.d/pptpd stop
/etc/init.d/pptpd start

6) Run /etc/rc.local

7) If you run a firewall on the server make sure to open port 1723. I do this with the following command:

# pptp VPN
/sbin/iptables -A extern -p tcp --dport 1723 -j RETURN

Notice that I am using a chain named "extern". This is probably not the case at your end.

8) Connect from your VPN client

MySQL JDBC Memory Usage on Large ResultSet

MySQL JDBC Memory Usage on Large ResultSet

Apache Velocity Servlets + UTF-8 = love

If you have problems outputting UTF-8 with Apache Velocity, create a file called velocity.properties under the WEB-INF folder and add the following 2 lines:

input.encoding=utf-8
output.encoding=utf-8

XML-RPC for PHP Library + UTF-8 = love

If you have problems with the XML-RPC for PHP library (http://phpxmlrpc.sourceforge.net/) and UTF-8, please be advised that in xmlrpc.inc you will find the following lines:

// The charset encoding used by the server for received messages and
// by the client for received responses when received charset cannot be determined
// or is not supported
$GLOBALS['xmlrpc_defencoding']='UTF-8';

// The encoding used internally by PHP.
// String values received as xml will be converted to this, and php strings will be converted to xml
// as if having been coded with this
GLOBALS['xmlrpc_internalencoding']='ISO-8859-1';

As you can see, internally everything seems to be converted to ISO-8859-1, no idea why. If you change this to
$GLOBALS['xmlrpc_internalencoding']='UTF-8';
you will be able to use UTF-8 correctly with the library.

MySQL + PHPMyAdmin + UTF-8 = love

Add the following to the [mysqld] section in /etc/mysql/my.cnf:

character_set_server = utf8
collation_server = utf8_general_ci
default-character-set=utf8

Adding AddDefaultCharset UTF-8 to /etc/apache2/conf.d/charset also helps with some web apps.

Tested on Ubuntu 8.04 Hardy