Category Archives: linux

Nginx and git via HTTP

Nginx is a high-performance HTTP server and proxy that has been gaining a lot of traction recently. It’s easy to set up, can handle a large number of clients with minimal resources, modular and supports SSL/TLS. Dynamic web content can be served via FastCGI to Nginx.

Git uses WebDAV for pushing via HTTP(S). Even though Nginx handles WebDAV requests, not all DAV commands are implemented in its DAV module. Locking and PROPFIND are missing in particular, and since these commands are used by Git when pushing via HTTP, pushing to a plain Nginx server is not possible at the moment.

One solution is to run Apache on a different port and proxy Git requests to it. However, Apache is such a resource hog that if a lightweight solution is desired it’s out of question. Here comes PyWebDAV, a standalone WebDAV server written in Python, into the picture.

Continue reading

Python-iptables

Python-iptables is a Python project that provides bindings to the iptables C libraries in Linux. Interoperability with iptables is achieved using the iptables C libraries (libiptc, libxtables, and iptables extensions), not calling the iptables executable and parsing its output as most other iptables wrapper libraries do; this makes python-iptables faster and not prone to parsing errors, at the same time leveraging all available iptables match and target extensions without further work. If you need further information on how iptables and netfilter work, this is a great summary.

Continue reading

How to use the latest 802.11 drivers and stack with Linux

There might be several reasons why the wireless drivers from your distro don’t fit the bill. You purchased a new wifi card which doesn’t have a driver in the kernel of your Linux system, or you would like to play with some cool new feature or just to make sure you have the latest and greatest, cutting edge 802.11 software.

Thanks to the work of the Linux wireless developers it is now possible to do that without upgrading your kernel. Using a compatibility layer you can compile and install the latest drivers and the 802.11 stack, and use it with your current kernel (that is, if you have a relatively recent kernel — if you have 2.6.24 or anything more recent, you’re good to go).

To use bleeding edge drivers, first make sure your kernel headers package is installed. Then fetch the necessary git trees:

mkdir compat-wireless && cd compat-wireless
git clone git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat-wireless-2.6.git

The linux-next git tree contains the code for the next kernel release, and 802.11 updates are pulled into it on a regular basis. The compat and compat-wireless-2.6 trees implement the necessary compatibility layer, thus making it possible to use the wireless subsystem from linux-next with your kernel. You can also use your git trees to follow development, and keep your wireless drivers up to date: just pull from the remote repositories when you’d like to update. However, the linux-next repository is rebased on a regular basis, so you may want to check out the daily tag instead of pulling.

You can use the compat-wireless-2.6 directory to actually build the wireless subsystem:

cd compat-wireless-2.6

Specify where your newly fetched kernel tree and the compat tree are:

export GIT_TREE=`pwd`/../linux-next/
export GIT_COMPAT_TREE=`pwd`/../compat/

Then refresh compat-wireless-2.6 with the contents of your git trees:

./scripts/admin-refresh.sh

You can also select just a specific driver or group of drivers, e.g. if you only need the ath5k driver:

./scripts/driver-select ath5k

And that’s it, everything should be ready for the build.

make
sudo make install

Your newly built modules will go into the updates folder inside your kernel module directory. Reboot your box (or use the scripts supplied with compat-wireless-2.6 to reload any drivers).

Disable scanning in NetworkManager when connected

Nowadays virtually all Linux distributions use NetworkManager to manage available network interfaces.  It makes interfaces connect automatically, for instance wireless network cards associate automatically to the access point with the strongest signal; ethernet cards can be set up via DHCP or configured with static parameters.

However, the strategy for keeping wireless interfaces connected involves scanning all available channels periodically — typically every two minutes.  For some wireless drivers, such as ath5k and ath9k this might take quite a few seconds, which means network traffic will stall for this period.  Even worse, while the problem is known, it is not possible to disable this feature or override the frequency of scans.  Background scanning is supposed to solve this issue, but the ath5k driver in Ubuntu Lucid does not support it yet.

Out of any other options, I decided to disable scanning in the source code and recompile NetworkManager.  It is quite trivial; as a first step, just fetch the source package:

% apt-get source network-manager

This will fetch and unpack the source tree of the network-manager package.

Update: you can install the build dependencies of network-manager via (thanks Andreas for the tip):

% apt-get build-dep network-manager

Save this patch, step into the network-manager folder, and apply the diff:

% curl -O http://nilvec.com/downloads/network-manager_disable-scan-if-connected.patch

% cd network-manager-0.8

% patch -p1 < ../network-manager_disable-scan-if-connected.patch

Update: to make sure Ubuntu won’t reinstall the original package you can bump the version of network-manager via (debchange is in the devscripts package):

% debchange -i

Just bump the version, or extend the version string with an extra part e.g. if you had 0.8-0ubuntu3, make it 0.8-0ubuntu3-disablescanning0.

% fakeroot dpkg-buildpackage

This patches and builds NetworkManager with scanning disabled while connected to an active access point.

% cd .. && sudo dpkg -i network-manager_0.8-0ubuntu3_i386.deb

There you go.  You should restart the network-manager service (or reboot your machine), and everything should work as supposed — no network stalls or lost packets due to wireless scans.

Building cyanogenmod from source

Cyanogenmod 5.0.7 stable has just been released for the Dream, Magic, Nexus One and Droid phones.  This is a great alternative to get Android 2.1 Eclair, especially to G1 (aka Dream) owners, who are still stuck with Android 1.6 — that is likely not going to change, due to the way the flash storage of the Dream is partitioned, not leaving enough room for a full 2.1 software update.

For the brave, it is also possible to build a Cyanogen ROM from source.  However, I haven’t been able to find any complete guide how to do that, so here’s one quick recap how it can be done.

It’s recommended to use Linux for the build process.  Ubuntu 8.04 or later will do.  You need the following packages:

sudo apt-get install git-core gnupg sun-java5-jdk flex bison gperf libsdl-dev libesd0-dev libwxgtk2.6-dev build-essential zip curl libncurses5-dev zlib1g-dev
This is for a 32-bit installation, on a 64-bit system they may have different names or you may have to install the 32-bit version from certain packages.  Ubuntu 9.10 and later doesn’t include Java5 from Sun anymore, but the Jaunty repositories can be used — just add the following lines to /etc/apt/sources.list:
deb http://us.archive.ubuntu.com/ubuntu/ jaunty multiverse
deb http://us.archive.ubuntu.com/ubuntu/ jaunty-updates multiverse
You also need the repo tool to fetch and manage Android source repositories:
curl http://android.git.kernel.org/repo > ~/bin/repo
chmod 755 ~/bin/repo
Once all the tools above are installed you can grab the Cyanogen Eclair repository:
mkdir cm
cd cm
repo init -u git://github.com/cyanogen/android.git -b eclair-ds
repo sync
This will clone all git repositories needed for the build.  It takes some time, so relax and have a coffee or two…  You can also use this time to install the Android SDK if you haven’t done so before, since you will need adb to access your phone.  Make sure the tools folder from the unpacked SDK is in your PATH.
Once you have the repositories you’ve got to extract the proprietary HTC files from your device.  Hook your phone up to your PC via USB.  Make sure your computer can see it:
% adb devices
List of devices attached
HT851N000044    device
Then you can go ahead and fetch the files:
cd vendor/htc/dream_sapphire
./extract_files.sh
If you don’t have your phone at hand you can also download an official HTC firmware and get the necessary files from it.  Go to htc.com and get signed-dream_devphone_userdebug-img-150275.zip.  You can use unyaffs to unpack system.img after you uncompressed the zip file, and then you can copy all necessary files from the folder it was extracted into.
Then go back to your build root, set up the build environment and choose your build target:
cd ../../..
source build/envsetup.sh
lunch
If you use zsh envsetup.sh may fail, but you can simply
unsetopt NOMATCH
in your shell and you’re good to go.  You can build everything with a simple
m
If you only need the OTA-format update zip use:
TARGET_NO_RADIOIMAGE=true make otapackage
One drawback of this process is that it produces an image that barely fits onto the G1 even after repartitioning it with the Danger SPL — there won’t be enough space left for Google applications like Gmail, Market and Talk.  I haven’t been able to find any clue how “official” Cyanogenmod images are built so that it remains slim enough to make Google’s applications also fit onto the phone — the xbin folder contains a squashfs image that spares some megabytes, but it is not enough in itself.  If you install this image you have just built, the Google applications won’t fit.  Perhaps the key is some build option which optimizes for size — if you figure it out, let me know.
Happy hacking!