Wednesday 3 June 2015

How to Build an Android ROM

What follows is more a simple explanation on how to build an Android ROM. I’m going to discuss several parts that can’t be absent if you want a fully functional Android phone, from the software point of view that is.
Here’s a brief summary :
  • Kernel
  • Libraries (and modules)
  • Bootloader
  • Recovery
  • Radio
  • Apps, framework, core, android-runtime, etc…

Kernel

A kernel is a critical component of every Operating System. It can be seen as a sort of bridge between the applications and the actual hardware of a device. Usually the data processing part is done at hardware level, furthermore the kernel is the most low-level abstraction layer for the resources.
There exist several types of kernels, but I am only discussing the one that is important for the Android ROM. The kernel for the Android ROM is a hybrid kernel, it is based on the Linux kernel. Devices can differ in RAM memory, ROM memory, hardware parts and so on. So it’s really important you have a kernel for your type of device, an HTC Wildfire kernel won’t work on a Nexus One for example.
It is possible to overclock a kernel. You may wonder, what the hell is overclocking? Every CPU is designed to work at a certain clock speed. For the HTC Wildfire the Qualcomm ARMv6 processor operates at 528 mHz — this is a limit that is coded in to the kernel because they can’t guarantee good cooling for the CPU if you go over this limit. My Wildfire runs at 652.8 mHz and it can go to 768 MHz until it freezes. When you overclock the kernel, the CPU will be instructed to do more calculations per second; so, it will increase performance.
When you overclock the CPU of your desktop machine, it will need extra cooling, but fortunately that’s not necessary for a smartphone. However, keep in mind that this will degenerate your CPU much faster than if it was at stock. To overclock an Android phone you must root it and install SetCPU or another overclocking app from the Play Store. Then you will have to flash a kernel that supports overclocking. I prefer the OC Kernel of HCDR.Jacob, at XDA-forums (see links right). If you have done these 3 things you’re ready to OC!

Libraries

Imagine you want to program an application for your phone that uses the camera. It can take a picture and turn the flash LED on. These are things not only your own application will use, but there can be tons of applications that have a button that can take a photo. So, imagine those apps always having to write the machine code to get the camera to turn on; it would be a lot of code duplication and would make an application slow. Instead, for functions like the camera or loudspeakers, they developed libraries. These are chunk of codes that can be executed by calling them trough a call-method (in java : import somelibrarie.someclass;). These are already pre-written and ready to use. It saves a lot of coding work and keeps the source code small. Libraries in the Windows OS for example are called .dll files (Dynamic Linked Libraries), whereas on your Android ROM or Linux OS these are the .so files (Shared Object). When coding in Java you import .jar libraries before the class signature. On your Android phone you have particular libraries that can’t be absent, or else your ROM won’t even boot.

Bootloader

A bootloader in general is the first bytes of code that a machine executes that will tell the bootsequence and will load the operating system into the RAM. On an Android phone this is the so called HBoot. From here you can go to the recovery, system or data partitionn.  By pressing the power button on your phone, HBoot will load the OS into RAM. By pressing the power and volume down buttons, you’ll bring up the HBoot menu.
The bootloader is used as an exploit for rooting an Android phone. Once rooted you can flash custom ROMs, like ImPrOS, or just edit system files (for a how-to, check tutorials under Android.)

Recovery

When you use a one-click-root application like Unrevoked3, it will install a modded recovery. With this modded recovery it is possible to flash a new kernel, radio image (see below), custom ROM, install applications (not the best way though), Nandroid backup, etc. The modded recovery that is flashed by Unrevoked3 is called ClockworkMod.
Recovery is a method for you to not only save yourself when you screw up (which you will! Tongue Out), but to update the ROM or kernel that’s on your phone, partition your SD card to use with a2sd, and many other OS-related things.  There’s a lot of different recoveries for different phones, so you’ll have to do some research of your own into what the most respected one is for your phone.  The recovery is essential, because you want to always have a fallback in case something bad happens to your phone’s software.

Radio

The lowest part of software layer is the radio: this is the very first thing that runs, just before the bootloader. This handles the GPS antenna, GSM antenna and fires up the CPU; everything that HBoot needs to load the OS. This can also be upgraded by flashing a new radio image through your recovery. However, this is not recommended: flashing a new ROM can’t brick your phone (make it unusable), but if anything goes wrong when you flash a ROM with a new radio image, this can brick your phone. So unless you experience bad reception or battery drainage, don’t touch the radio!
Posted by at No comments:

Building a Kernel from source

What is a Kernel?

The operating system of a device is the part of the device responsible for basic use and administration. This includes the kernel and device drivers, boot loader, command shell or other user interface, and basic file and system utilities. Whereas the user interface is the outermost portition of the operating system, kernel is the innermost. It is the core internals, the software that provides basic services for all other parts of the system, manages hardware and distributes system resources.
Typical components of a kernel are interrupt handlers to service interrupt requests, a scheduler to share processor time among multiple processes, a memory management system to manage process address spaces, and system services like networking and inter-process communication. On modern systems with protected memory management units, the kernel typically resides in an elevated system state as compared to normal user applications. This includes a protected memory space and full access to hardware. This system state and memory space is collectively referred to as kernel-space. Conversely, user applications reside in user-space.
Applications running on the system communicate with the kernel via system calls. An application typically calls functions in a library-Eg The C library–that in turn rely on the system call interface to instruct the kernel to carry out tasks on the application’s behalf.
Our central theme is of course the Android device (a phone, or tablet or any other device), and here, Android is the Operating System. An Android Kernel is essentially a modified Linux Kernel with specific modifications to support the device architecture. I won’t bore you any further with the theoretical aspects of a kernel, and if you’re interested in knowing more about a kernel, read the book, Linux Kernel development, authored by Robert Love.
Without much further ado, let me jump into the topic proper, which is about how to compile an Android Kernel. For this, you’re expected to be familiar with the Linux command line, and know some basic file copying and moving commands. Though a working knowledge of Git would be beneficial, in this tutorial, I will be feeding you with the essential commands on a platter.
I will describe compilation of the HTC Desire Kernel which works with Gingerbread Sense Roms. However the process applies with minor modifications to any Android Device and kernel. The specifics may be variable in how the kernel gets packaged, or flashed.

What you need:

  1. A Linux installation (a PC on which a linux distro is installed) or Linux box (May be a live cd or vmware like box). Any linux distro will do, but I assume an Ubuntu installation to simplify the explanation.
  2. A toolchain-Either the Android NDK, or your own toolchain
  3. Android kernel source code for your device. This tutorial descibes instructions for both the Htc Desire and Samsung Galaxy Note N7100. With minor differences, the method is practically the same for any Android device.
  4. Familiarity with the linux shell and basic linux commands.
Each step will be explained as we proceed, so if terms like “toolchain” bother you, don’t worry. It will be explained in detail later. Even if you’re a linux newbie, don’t worry. Almost all commands you need to master this tutorial will be dictated to you. Just make sure you either copy and paste the commands exactly as described here, preserving case (meaning you shouldnt type out “Cp” when the command is given as “cp”. Linux shells are case sensitive.

Introduction to the command line for newbies:

As you proceed through this tutorial, you will see various commands in boxes, which may be typed out on your command shell, preserving case and spaces as they appear. As an example, the following is given:
In this case, the code in the box is a command which should be typed out on the shell (or console).
Common mistakes include typing the following:
(i) LS -L ~/android
wont work as LS and ls are different
(ii) ls -l ~\android
On Linux, “\” the backslash has no role on the command line (An exception is its use in regular expressions and strings, as a means of escaping characters. This won’t be described in this tutorial though, as it’s rather out of scope of our discussion). You need to use “/” or the forward slash.
(iii) ls -l ~\ANDROID
won’t again work, as ANDROID and android are two different files (or directories).
Ok, now that you have a feel for what it’s like, using the linux console, let’s begin the tutorial proper.

1. Getting the source code

At this point, you need to download the source code for your kernel. There are generally, two ways to get kernel source code:
(a) From a compressed archive uploaded by the manufacturer of the device.
(b) From a git repository of another developer.
The tutorial will use the first method.
The HTC Desire source is available from two kinds of resources-you can either get it from htcdevs.com (official HTC Dev site), or from source code uploaded from someone else. For the purpose of this tutorial, I’ll assume we’re working on the official HTC GB source code. So download bravo_2.6.35_gb-mr.tar.gz from htcdevs.com.
In case, you’re working on a Samsung kernel, you can get your source code here.
In many cases, you may find it much easier to reuse another developer’s source code. For this, visit their XDA kernel thread, and search for instructions regarding where they’ve shared their source code. As an example of this method, let’s look at developer g.lewarne‘s source code. His kernel is titled Note2Core Kernel for Galaxy Note II N7100 / N7105 (LTE), and can be found here. If you read the thread, you will see that he has shared his source code at github here. I will describe how to use this, later.

2. Setting up the host PC and preparing source code

2.1 Install some essential linux packages from the Linux terminal:

2.2 Extract the source code

The file you downloaded is a tar archive (like a zip file), so you need to extract it to a convenient location. Let’s hit the linux shell-open a terminal window in linux (Accessories->Terminal)
Let’s start off in our home directory:
Now, create the directories for our kernel compilation box:
Now you need to copy the tar.gz file from wherever you downloaded it to, to this dir. You can use a file explorer GUI like Nautilus or Dolphin.
Extract the archive:
Now we can view the extracted files within the directory: ~/android/kernel/bravo_2.6.35_gb-mr/

2.3 Set up the toolchain

A toolchain is a set of programs which allow you to compile source code (any source code, not just kernels). The toolchain is specific for the processor and hardware, so we need a toolchain specific for Android and especially the Desire. If you’re a semiadvanced-pro user, you may consider compiling your own toolchain (See theGanymedes’ guide for doing so). If compilation of kernels is all that you require, fortunately for you, there is an easy way-the Android NDK – v7 (latest as of now) is available here
Get the NDK for Linux – android-ndk-r7-linux-x86.tar.bz2
Now copy the NDK file to: ~/android/ndk
Whenever I say copy, you have to manually copy the file with any file manager. Nautilus comes with Ubuntu, and Dolphin with Kubuntu. You may also use the shell of course with
Extract it:
Now add the path for your toolchain to the env variable:
At the end of the file, add this line:
Code:

3. Setting up kernel parameters

Kernels are compiled with a program called gnu make, and use a set of configuration options specified within a file called Makefile.
A vital point to note is that kernels are compiled with a program called gcc (basically the gnu C compiler), and our NDK itself has its own optimized version of gcc. While compiling, we’re actually cross compiling it (meaning compiling a binary package on a system which is different from the actual system which is meant to run it- you’re compiling it on your PC while it’s actually meant to run on your Desire)
This means that when you compile it, you have to make sure that you compile it with the NDK’s version of gcc instead of the system version. Otherwise you end up with a kernel meant to run on your pc, duh! Specifying which gcc to use is by the CROSS_COMPILE variable. You can set it up with this command:
Code:
Note the hyphen (-) at the end, and do not forget to include it! At compilation time, system will actually use this variable to find all the programs it needs. Eg: The path for gcc will become arm-linux-androideabi-gcc
We can compile kernels with many different options, like with ext4 support, or without; ext4 support as part of the kernel zImage (in which case it makes the kernel larger), or as a loadable module (of the form somename.ko, which is loaded at init.d/init.rc with the command insmod modulename.ko)
We specify the exact options we require with the help of a useful configuration program called menuconfig (which as the name suggests, is a menu for configuration of make options).
An important thing to note is that as far as kernel compilation is concerned, there are a vast amount of options to setup, and unless you’re thorough with kernel compilation, you wont be able to set up the options correctly and get your kernel to boot. Fortunately for us, the kernel source already comes with a default set of parameters which can be easily set up.
Note that all make commands must be executed within the directory bravo_2.6.35_gb-mr. Let’s go there now:
This produces a .config file (used by the menuconfig) containing essential parameters to produce a booting kernel for the Desire.
In case you’re not working on the HTC Desire, but a different device, you need to run the defconfig of your device. For this, you need to know the name of the script which runs defconfig. You can get the name by inspecting the names of the files in [kernel source folder]/arch/arm/configs. In fact each file located there is a renamed .config file. Eg: For the Note 2, you will find the file t0_04_defconfig. So to run a defconfig for Note2, you would type:
Note: There is a simpler way to get the basic .config file, and this is to get it from a running kernel built by someone else. You can extract the .config from a running kernel with these commands:
Note that not every kernel include a config.gz. Your kernel developer needs to have included this option while compiling his kernel
Now we can open menuconfig and add anything we need in addition:
Advanced: You can view the huge amount of options available in menuconfig, pick and choose the ones you want. As a general word of caution, in your initial compilation, you shouldnt try to modify anything. Just pick the default .config, and compile once. If it succeeds, try booting your system with the kernel, and any default modules like the Wifi module. Once you’ve confirmed that it boots without any bootloops or hangups, you should enable Wifi and confirm that it works. This two stage check is a confirmation that you can compile a default (or stock) kernel, and also successfully compile modules. Once you can, save your .config to a safe location. If you’re using a git version control system (which you should), at this stage, you could also commit your changes, and label them as “Initial commit”. Now, you can go ahead and add changes to the kernel.
You can add ext4 support for example.
Once you’re done choosing options, you can exit menuconfig.
Tip: Once you’ve done menuconfig once, and find that you’re always using ARCH=arm and the same CROSS_COMPILE prefix regularly, you can hardcode these options into the main Makefile so you can just type ‘make’ each time for compiling the kernel. Another option is to “export” the prefices before compiling.
Advanced: In fact, once you’ve made your first successful compile, edit your main Makefile (the Makefile that resides in the root folder of the kernel source tree), and change the CROSS_COMPILE variable to point to your toolchain. The Makefile also has a variable for ARCH, which by default is arm. Once you set both of these, you can compile by simply executing:

4. Compiling it

This is simple. The basic command is:
The -j4 specifies the number of jobs to execute per operation. This is typically equal to the processor cores in your system. If you’re on a VPS with a fair usage policy, you had better keep it a notch below the maximum allowable CPU load.
During compilation, you will see all sorts of messages, which may include warnings too. In most cases, its safe to ignore warnings. If there are errors, the compilation will stop, and you will have to fix the issues.
If you’ve modified your Makefile to include ARCH and CROSS_COMPILE variables as described in the previous section, you can compile simply by running:

5. Distributing your kernel to users

At the end of compilation, it generates files named zImage, and various .ko files.
You have to copy them from their default location to a zip file. The best way is to use my variant of koush’s Anykernel, and copy the files to it. Then, you can zip the whole folder and lo and behold-you have your flashable kernel zip which you can distribute to others.
You can also remove the zImage and the modules from /system/lib/modules of any kernel zip available with you, and copy over your files to it, at the correct location.
So, let’s say that you have extracted an existing kernel zip to the location ~/flashable, then the file structure should be like this:

I’ll include my flashable zip directory along with this post. Download file kernel_flashable.tar.bz2.zip to ~/
(Note: I’ll include this file later)
This will create the directory structure outlined above.
Now after every compilation of the kernel, execute these commands from where you executed make:
This will create mykernel.zip at ~/kernel_flashable. You can distribute this to your users to flash. Make sure you edit updater-script before though.

6. Some kernel compilation errors:

Treat warnings as errors-Solved by removing the string “-Werror” from all Makefiles of the file which failed to compile. Some people had said that the real error (Array out of bounds warning) was because of gcc optimizations. But putting -O2 to -O0 didnt do a thing.
No of jobs – ought not to exceed 50.
“warning: variable set but not used [-Wunused-but-set-variable]”-Look at KBUILD_CFLAGS in the main Makefile. Add -Wno-error=unused-but-set-variable to the existing set of flags.
Note the following from gcc manual:
-WerrorMake all warnings into hard errors. Source code which triggers warnings will be rejected.
-w Inhibit all warning messages. If you’re familiar with C code and like to fix stuff, rather than ignoring potential bugs, use this only as a last resort- A ‘brahmastram’ (most powerful weapon in your time of gravest need) as the epics would say.
-WerrorMake all warnings into errors.
-Werror=Make the specified warning into an error. The specifier for a warning is appended, for example -Werror=switch turns the warnings controlled by -Wswitch into errors. This switch takes a negative form, to be used to negate -Werror for specific warnings, for example -Wno-error=switch makes -Wswitch warnings not be errors, even when -Werror is in effect. You can use the -fdiagnostics-show-option option to have each controllable warning amended with the option which controls it, to determine what to use with this option.
So what I did to suppress errors was to add:
Though the -Wunused-but-set-variable is not a real issue in itself, it generates so much “noise” that you may miss actual make errors.
This is the error what I was talking about:
Solution:
Edit drivers/net/wireless/bcm4329_204/Makefile
Locate -Werror within DHDCFLAGS, and delete it.
This will prevent gcc from treating mere warnings as errors.

7. Modifying Kernel source code on the fly – Applying Kernel Patches

Ok, you have compiled a simple stock kernel. Now what? Would you like to add fixes/mods developed by other kernel devs? This post explains patches and how exactly to do this.
Patches to the kernel are applied via patch files. Patch files are simple text files generated by the linux diff program which takes two text files, compares them and writes the differences (hence called diff) to another text file which by convention has the extension .patch

7.1 Example patch

Following is a patch containing my “Extended battery” fix with Sibere’s battfix. I’ll explain patching with this. Let’s understand the patch file. Open it up in any text editor.
Note the first line:
diff -rupN basically describes the command that was used to generate this patch. The -u means that the patch file is something called a universal patch
bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c was the original file, and bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c was the target file or file which contains the mod.

7.2 How to apply patch files

The command depends on where your current directory is. If you’re in ~/android/kernel/bravo_2.6.35_gb-mr/ and your current directory contains the directory ‘drivers’, you can apply this patch with this command:
If you’re within drivers, then you have to modify the command like this:
Hope you get the gist. Basically, as you move into the source tree, you have to increment the patch level by the number of directories you’ve moved down into. Very simple, isnt it?

8. Sharing and Collaborating – Using Github and Commits

Kernel compilation is a group effort (at least it ought to be). When different devs work on different parts of the code and create their own mods, development progresses. For this purpose, it is important that you share your code with other devs. The best way to do this to upload your sources to github.
First, create a github account.
Next you can view other devs’ github sources and examine their commits. Commits are basically patches applies to the previous source uploaded. Github commits use the universal patch format and can be viewed directly, downloaded as patch files, and applied to your code. You can also choose to download the whole source tree uploaded by another dev and examine it.
Posted by at No comments:
Subscribe to: Posts (Atom)