Author Archives: ldx

How to use the latest 802.11 drivers and stack with Linux

There might be several reasons why the wireless drivers from your distro don’t fit the bill. You purchased a new wifi card which doesn’t have a driver in the kernel of your Linux system, or you would like to play with some cool new feature or just to make sure you have the latest and greatest, cutting edge 802.11 software.

Thanks to the work of the Linux wireless developers it is now possible to do that without upgrading your kernel. Using a compatibility layer you can compile and install the latest drivers and the 802.11 stack, and use it with your current kernel (that is, if you have a relatively recent kernel — if you have 2.6.24 or anything more recent, you’re good to go).

To use bleeding edge drivers, first make sure your kernel headers package is installed. Then fetch the necessary git trees:

mkdir compat-wireless && cd compat-wireless
git clone git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat-wireless-2.6.git

The linux-next git tree contains the code for the next kernel release, and 802.11 updates are pulled into it on a regular basis. The compat and compat-wireless-2.6 trees implement the necessary compatibility layer, thus making it possible to use the wireless subsystem from linux-next with your kernel. You can also use your git trees to follow development, and keep your wireless drivers up to date: just pull from the remote repositories when you’d like to update. However, the linux-next repository is rebased on a regular basis, so you may want to check out the daily tag instead of pulling.

You can use the compat-wireless-2.6 directory to actually build the wireless subsystem:

cd compat-wireless-2.6

Specify where your newly fetched kernel tree and the compat tree are:

export GIT_TREE=`pwd`/../linux-next/
export GIT_COMPAT_TREE=`pwd`/../compat/

Then refresh compat-wireless-2.6 with the contents of your git trees:

./scripts/admin-refresh.sh

You can also select just a specific driver or group of drivers, e.g. if you only need the ath5k driver:

./scripts/driver-select ath5k

And that’s it, everything should be ready for the build.

make
sudo make install

Your newly built modules will go into the updates folder inside your kernel module directory. Reboot your box (or use the scripts supplied with compat-wireless-2.6 to reload any drivers).

Testing an Android image built from source

So you just built a new android system from source and you would like to test it first in the emulator before flashing it onto your phone.  Or perhaps you would just like to know how the bleeding edge version of android straight from the source repositories looks.  Fortunately, the emulator used by android SDK is flexible enough, so you can specify what image it should use for the data, system or the other partitions — thus you can build your own flavor of android and test it in the emulator easily.

Once you have checked out the android source tree via repo (either the official one, cyanogenmod or some other mod which published its source via a manifest file) apply your changes if any (add a new component to the system, modify the framework etc).  Then just pick a build target (it can be for example full-eng, it will work with the emulator) and build it the usual way.  By default the qemu-based emulator is also built, and it will be placed to out/host/linux-x86/bin/.  When it’s done the images will be in out/target/product/(your target device name)/.  From now on I’ll assume you picked the generic target.

You will also likely need an SD card image that can mounted by the emulator.  The SDK contains a tool, mksdcard with which it’s simple enough to create one:

$ mksdcard 1024M out/target/product/generic/sdcard.img

The SDK tools/ folder should be in your $PATH for this to work.

Now the only thing left is to fire up the emulator with your newly built images:

$ ./out/host/linux-x86/bin/emulator \
-sysdir out/target/product/generic \
-kernel prebuilt/android-arm/kernel/kernel-qemu \
-data out/target/product/generic/userdata.img \
-sdcard out/target/product/generic/sdcard.img

That’s all, the emulator should start up, and if you are lucky your new android hack will come to light and work.  You can use adb to copy files to/from the emulator image, or just fire up a root shell in the emulator via adb shell and take a look what is going on.

IndentFinder vim plugin

This is a great little python script that finds out what indentation is used in your source files. The author also provides a vim plugin, which is a life saver for anyone working with others on software projects. It prevents the complete mess that is the result of team members using their own indentation settings on source code. It analyses the text file, and provides reliable hints what indentation settings (tabs vs spaces, number of spaces etc) were used by the author. There is another plugin for the same purpose, which is basically the vimscript port of IndentFinder, but this latter is extremely slow, especially on large files (it seems vimscript was not really meant for parsing thousands of lines every time you open a file).

What I did not really like about IndentFinder was the way it invokes the python interpreter (thus increasing startup time and leaving room for errors), even if python bindings are supported in vim.  One solution could have been simply to create a vim plugin from the python source, but then this would have meant increased maintenance efforts (copying changes from the python source to the vim plugin).  Another possibility was to generate the vim plugin via setup.py from the python source.

First I added a new method to the main class in indent_finder.py:

--- a/indent_finder.py
+++ b/indent_finder.py
@@ -125,16 +125,21 @@ class IndentFinder:
         self.clear()^M
         f = open( fname )^M
         l = f.readline()^M
         while( l ):^M
             self.analyse_line( l )^M
             l = f.readline()^M
         f.close()^M
 ^M
+    def parse_buffer( self, buf ):^M
+        self.clear()^M
+        for l in buf:^M
+            self.analyse_line(l)^M
+^M
     def clear( self ):^M
         self.lines = {}^M
         for i in range(2,9): self.lines['space%d' % i] = 0^M
         for i in range(2,9): self.lines['mixed%d' % i] = 0^M
         self.lines['tab'] = 0^M
 ^M
         self.nb_processed_lines = 0^M
         self.nb_indent_hint = 0^M

It just makes it possible to call the parser with a buffer instead of a file name: vim opens the file and reads its contents into memory, thus it’s easier to pass the buffer itself to IndentFinder.

The second part is modifying setup.py to build the plugin from a fixed header, the python source and a fixed trailer. It turned out distutils is flexible enough to do this easily: you can subset the build class and do your special stuff in it. Easy:

#!/usr/bin/env python

from distutils.core import setup
from distutils.command.build import build
import os
import indent_finder

HEADER = """if exists('indent_finder') || !has('python')
	finish
endif
let indent_finder=1

python << EOS
"""

TRAILER = """EOS

augroup indent
	au!
	au BufRead *.* :py import vim
	au BufRead *.* :py fi = IndentFinder()
	au BufRead *.* :py fi.parse_buffer(vim.current.buffer)
	au BufRead *.* :py vim.command(fi.vim_output())
augroup END
"""

These are our fixed header and trailer parts. We just make sure python is supported in vim, and make IndentFinder parse the opened buffer. The result from IndentFinder is an actual vim command that adjusts indentation settings.

class build_vimscript(build):
	vimscript = "indent_finder.vim"
	pyscript = "indent_finder.py"

	def run(self):
		build.run(self) # call superclass first

		if self.is_vimscript_outdated():
			self.update_vimscript()

	def is_vimscript_outdated(self):
		pyst = os.stat(self.pyscript)
		try:
			vimst = os.stat(self.vimscript)
		except OSError:
			return True
		else:
			if pyst.st_mtime > vimst.st_mtime:
				return True
		return False

	def update_vimscript(self):
		pyf = open(self.pyscript, "r")
		vimf = open(self.vimscript, "w")

		vimf.write(HEADER)
		for line in pyf:
			vimf.write(line)
		vimf.write(TRAILER)

		vimf.close()
		pyf.close()

Here we check that the vim plugin is older than the python source of IndentFinder. If it is, then the plugin is rebuilt. Calling the superclass in our run() method makes sure the other part of the build process is handled the usual way.

setup( 	name = "Indentation Finder",
		version = indent_finder.VERSION,
		description = "Finds of check the indentation used in programming source files",
		author = "Philippe Fremy",
		author_email = "phil at freehackers dot org",
		maintainer = "Philippe Fremy",
		maintainer_email = "phil at freehackers dot org",
		license = "BSD license",
		url="http://www.freehackers.org/Indent_Finder",
		py_modules = [ "indent_finder" ],
		cmdclass={"build": build_vimscript},
		)

Here we’ve just added a new parameter to setup cmdclass tells distutils to use our class for building the package.

And that’s it. The .vim plugin is built and can be copied to
~/.vim/plugin/.

Disable scanning in NetworkManager when connected

Nowadays virtually all Linux distributions use NetworkManager to manage available network interfaces.  It makes interfaces connect automatically, for instance wireless network cards associate automatically to the access point with the strongest signal; ethernet cards can be set up via DHCP or configured with static parameters.

However, the strategy for keeping wireless interfaces connected involves scanning all available channels periodically — typically every two minutes.  For some wireless drivers, such as ath5k and ath9k this might take quite a few seconds, which means network traffic will stall for this period.  Even worse, while the problem is known, it is not possible to disable this feature or override the frequency of scans.  Background scanning is supposed to solve this issue, but the ath5k driver in Ubuntu Lucid does not support it yet.

Out of any other options, I decided to disable scanning in the source code and recompile NetworkManager.  It is quite trivial; as a first step, just fetch the source package:

% apt-get source network-manager

This will fetch and unpack the source tree of the network-manager package.

Update: you can install the build dependencies of network-manager via (thanks Andreas for the tip):

% apt-get build-dep network-manager

Save this patch, step into the network-manager folder, and apply the diff:

% curl -O http://nilvec.com/downloads/network-manager_disable-scan-if-connected.patch

% cd network-manager-0.8

% patch -p1 < ../network-manager_disable-scan-if-connected.patch

Update: to make sure Ubuntu won’t reinstall the original package you can bump the version of network-manager via (debchange is in the devscripts package):

% debchange -i

Just bump the version, or extend the version string with an extra part e.g. if you had 0.8-0ubuntu3, make it 0.8-0ubuntu3-disablescanning0.

% fakeroot dpkg-buildpackage

This patches and builds NetworkManager with scanning disabled while connected to an active access point.

% cd .. && sudo dpkg -i network-manager_0.8-0ubuntu3_i386.deb

There you go.  You should restart the network-manager service (or reboot your machine), and everything should work as supposed — no network stalls or lost packets due to wireless scans.

Building cyanogenmod from source

Cyanogenmod 5.0.7 stable has just been released for the Dream, Magic, Nexus One and Droid phones.  This is a great alternative to get Android 2.1 Eclair, especially to G1 (aka Dream) owners, who are still stuck with Android 1.6 — that is likely not going to change, due to the way the flash storage of the Dream is partitioned, not leaving enough room for a full 2.1 software update.

For the brave, it is also possible to build a Cyanogen ROM from source.  However, I haven’t been able to find any complete guide how to do that, so here’s one quick recap how it can be done.

It’s recommended to use Linux for the build process.  Ubuntu 8.04 or later will do.  You need the following packages:

sudo apt-get install git-core gnupg sun-java5-jdk flex bison gperf libsdl-dev libesd0-dev libwxgtk2.6-dev build-essential zip curl libncurses5-dev zlib1g-dev
This is for a 32-bit installation, on a 64-bit system they may have different names or you may have to install the 32-bit version from certain packages.  Ubuntu 9.10 and later doesn’t include Java5 from Sun anymore, but the Jaunty repositories can be used — just add the following lines to /etc/apt/sources.list:
deb http://us.archive.ubuntu.com/ubuntu/ jaunty multiverse
deb http://us.archive.ubuntu.com/ubuntu/ jaunty-updates multiverse
You also need the repo tool to fetch and manage Android source repositories:
curl http://android.git.kernel.org/repo > ~/bin/repo
chmod 755 ~/bin/repo
Once all the tools above are installed you can grab the Cyanogen Eclair repository:
mkdir cm
cd cm
repo init -u git://github.com/cyanogen/android.git -b eclair-ds
repo sync
This will clone all git repositories needed for the build.  It takes some time, so relax and have a coffee or two…  You can also use this time to install the Android SDK if you haven’t done so before, since you will need adb to access your phone.  Make sure the tools folder from the unpacked SDK is in your PATH.
Once you have the repositories you’ve got to extract the proprietary HTC files from your device.  Hook your phone up to your PC via USB.  Make sure your computer can see it:
% adb devices
List of devices attached
HT851N000044    device
Then you can go ahead and fetch the files:
cd vendor/htc/dream_sapphire
./extract_files.sh
If you don’t have your phone at hand you can also download an official HTC firmware and get the necessary files from it.  Go to htc.com and get signed-dream_devphone_userdebug-img-150275.zip.  You can use unyaffs to unpack system.img after you uncompressed the zip file, and then you can copy all necessary files from the folder it was extracted into.
Then go back to your build root, set up the build environment and choose your build target:
cd ../../..
source build/envsetup.sh
lunch
If you use zsh envsetup.sh may fail, but you can simply
unsetopt NOMATCH
in your shell and you’re good to go.  You can build everything with a simple
m
If you only need the OTA-format update zip use:
TARGET_NO_RADIOIMAGE=true make otapackage
One drawback of this process is that it produces an image that barely fits onto the G1 even after repartitioning it with the Danger SPL — there won’t be enough space left for Google applications like Gmail, Market and Talk.  I haven’t been able to find any clue how “official” Cyanogenmod images are built so that it remains slim enough to make Google’s applications also fit onto the phone — the xbin folder contains a squashfs image that spares some megabytes, but it is not enough in itself.  If you install this image you have just built, the Google applications won’t fit.  Perhaps the key is some build option which optimizes for size — if you figure it out, let me know.
Happy hacking!