Current Status of Libbash

Glad to see that libbash is accepted by Google Summer of Code this year. Congratulations to André Aparício! Petteri and I will both mentor this project during the summer. Before GSoC begins, I’d like to explain the current status of this project.

I worked on this project in GSoC 2011. After the summer ended, I spent months seeking a job so I didn’t pay much attention to the project at that time. And later, I made several commits to the project and it’s looking better.

So far, we can generate correct metadata for 13849 ebuilds (5435 more than before). Nearly half of the ebuilds in the main tree can be handled by libbash now. So the goal of this year’s GSoC is to make libbash be able to handle the rest of the ebuilds.

What’s stopping us from handling more ebuilds? There are several eclasses that our grammar file can not parse. That means any ebuild that inherits from these eclasses can not be handled by libbash. On the other hand, the runtime part is not fully implemented, which is due to lack of grammar support or low priority. I have to say it’s hard to tell all the problems we are facing at once. I have been working in this pattern for quite some time:
1. Use our implementation of instruo to generate metadata for the ebuilds in the main tree.
2. Sort the error output to see which errors are the biggest enemies.
3. Solve the problems that are figured out from the error output
Generating ebuild metadata will take about 10 minutes so this process is not fast. I need to solve as many problems as I can at one pass. Typically new errors will occur after I fix the old ones. And the new errors are not introduced by the new commits. They were just hidden by the old ones. That’s why it’s hard to move forward and handle more ebuilds in a short time.

In future, the error output should be improved so it would be easy for us to figure out more problems. After constantly working on the error output for some time, I’m sure we will get fewer errors and handle more ebuilds. Looking forward to seeing a working libbash for all the ebuilds at the end of this summer.

Ifnet updates for NetworkManager 0.9

NetworkManager 0.9 has changed a lot of stuff. Before NM 0.9, there are user connections and system connections. Ifnet plug-in is made to only take charge of system connections. But after NM 0.9, all connections become system connections and ifnet tries to manage all of them. Then some bad things might happen. The problem is reported by David Narvaez and he helped a lot on it. Bug #392409 is also caused by the problem. This is fixed in networkmanager-0.9.2.0-r2: if a user does not check “available to all users”, ifnet will not handle the connection any more and the default plug-in “keyfile” will do the job.

Another important change is that ifnet now supports openrc style. As bash arrays are not allowed by openrc, ifnet caused Bug #369711. After networkmanager-0.9.2.0-r2, ifnet can read both the old baselayout style (with bash arrays) and openrc style (without bash arrays), but it only writes to /etc/conf.d/net with openrc style.

A brilliant way to parallize your commands

The solution I’m taking about here comes from stackoverflow.com. The original question is :

I have a list/queue of 200 commands that I need to run in a shell on a Linux server. I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer. When a process finishes I want the next command to be “popped” from the queue and executed.

Solution: write a makefile with the following content:

all: usera userb userc....
usera:
imapsync usera
userb:
imapsync userb
....

Then just run

make -j 10 -f makefile

I think the answer is really brilliant as it not only makes use of available tools that are quite mature, but also solves the problem fast with little effort. If the situation is getting more complicated, the solution can still be easily adjusted. xargs is also a good solution btw.

libbash final report

My GSoC project will end in this weekend so this is the final report. This week, I spent most of my time making sure the project is in a good state for future developers.

So far, libbash can generate correct ebuild metadata for 8414 out of 27586 ebuilds. What is blocking us from moving on? We need to be able to parse all the eclasses. But there are still a few problems in the parser grammar. On the other hand, our runtime is not complete. In one word, more time is required. This is the ohloh page for libbash where you can learn some statistics of the code.

I wrote an ebuild for libbash and put it in my overlay. Now you can play with it by

layman -a qiaomuf
emerge libbash

Although libbash is not powerful enough to generate all ebuild metadata, it can do something useful now. As I said before, you can always check what libbash can support from these test scripts. One thing you should notice is that we do not support running external command.

The library is quite easy to use. To give you a taste, here is a code snippet showing how to use it:

#include <algorithm>
#include <iostream>
#include <iterator>

#include <libbash/libbash.h>

int main(int argc, char** argv)
{
  // store variable values and function names
  std::unordered_map<std::string, std::vector<std::string>> variables;
  std::vector<std::string> functions;

  // interpret the script specified by argv[1]
  libbash::interpret(argv[1], variables, functions);

  // do what ever you want with variables and functions
  std::copy(functions.begin(),
            functions.end(),
            std::ostream_iterator<std::string>(std::cout, " "));
  std::cout << std::endl;
}

To get it compiled, run

g++ -std=c++0x $(pkg-config --cflags --libs libbash) test.cpp

The program will interpret the script like bash and store the result in the variable map and function name list. All the variables are treated as arrays. For more information, you can check the API documentation.
(P.S.: The library is not stable so please don’t use it in any serious place)

So what’s next? I think it would be improving the parser grammar, figuring out problems from the output of instruo and implement more runtime. People are welcome to make contributions. I’ll continue making contributions after GSoC.

Last but not least, I’d like to thank Google and Gentoo community for giving me such great experience. Thank Petteri for mentoring me. I enjoyed a wonderful summer with you and learned a lot including C++0x, boost libraries, ANTLR, autotools, Scrum and so on. Thank Donnie, Robin and other people who helped me.

OK, I’m going to get myself prepared and seek a job 😉

libbash weekly report #11

Sorry for the delay this time because I was in Beijing in the last weekend. Finally I can get down to write the report.

Now 8414 ebuilds can be handled. The number is increasing again and it’s not that hard to make progress this time because we have fixed the big issues in the parser grammar. However, running instruo is not that fast any more. It is because we can parse more eclasses so there are more things to handle.

I did more than expected in the last week. Here is the list of what I have done in the last week:

• Fix code so that our instruo won’t crash anymore(turns out to be a problem related to thread-safety)
• Get libtool.eclass fully parsed
• Get perl-module.eclass fully parsed
• Get distutils.eclass fully parsed
• Get java-utils-2.eclass fully parsed
• Get kde4-base.eclass fully parsed
• Get python.eclass parsed with one exception: double quoted arithmetic expansion

The above eclasses included all the tasks for grammar improvements that I mentioned in the last weekly report.

This week I will:

• Improve our build system to install something useful
• Improve our ebuild so that the library can be installed and used properly
• Review all the documentation
• Review all the code

libbash weekly report #10

Finally I have done with the backtracking removal. Now doing semantic predicate is much easier. With this change, I successfully supported here document, which was the biggest blocker before I started this work. In addition, the performance is better. I used valgrind to get performance comparison. Here is the output of ms_print (post-processing tool for Massif) before and after backtracking removal: before, after. I reduced about 38% memory usage and got the library run 20% faster.

What I have done in the past week:

• Completed the work on the new parser and incorporated it to our project
• Improved parameter expansion parsing and its runtime
• Improved built-in and keyword test
• Improved the runtime for case statement
• Improved arithmetic expansion
• Improved the local built-in
• Fixed some minor problems in compound statement and parameter expansion
• Reimplemented the export built-in
• Removed several tokens to avoid conflicts
• Improved here document and here string
• Fixed single quoted string in command substitution

In the following week, I will:

• Get backtracking removal pushed
• Fix our instruo implementation (now it crashes with the new grammar)
• Improve process substitution
• Support redirection without any command
• Reimplement the local built-in
• Fix some minor problems in variable expansion and bash test
• Remove some composite tokens

I started working on this project early and I will start seeking a job soon (I’ll soon graduate from my university). So Petteri and I agreed to end the GSoC on 08.06. As a result, this is the last iteration of this year’s GSoC. I’ll write one more regular report and a final report before the end. I’ll continue my work on this project as soon as I get a job :).