Archive for August, 2010

Omni Completion in Vim

Posted: August 26, 2010 in C/C++, Programming, Vim
Tags: , , ,

Vim 7 onwards, we have Omni Complete feature. This feature is similar to IntelliSense in Visual Studio or other IDEs. It can show a pop up menu displaying the structure, class or scope context. Here is a screenshot of Omni Complete displaying the class scope in CCCC code base .

OmniCppComplete screen shot in CCCC code.

As you can see, it can show all methods and variables including its original context like type, prototype (in case of method), documentation (if documented), private/protected/public, file defined in etc.

For getting Vim to work like this, you will need

Install all 4 by whatever means (apt-get, yum, build from source). OmniCppComplete and SuperTab are Vim scripts, detailed installation steps are given in the respective links. GNU has a version of ctags. But what we need is exuberant ctags which is different from GNU ctags. Most popular Linux distros bundle GNU ctags, but exuberant ctags should be available from the repository.

Vim is an excellent editor, but does not understand C/C++ natively. This is where ctags comes in to the picture. Ctags will generate a “tags” file which is like an index of the project source code. This tags file is used by Vim and OmniCppComplete script to provide contextual data in a pop-up menu.

To build a ctags tags file, give the command from the top directory of the project. This can be done from any subdirectory too (for even more focused tags file).

ctags -R --languages=C,C++ --c++-kinds=+p --fields=+iaS --extra=+q ./

Run vim from the same directory. By default, Vim will search for tags file beginning from current directory to all parent directories. First tags file found will be used. For better usability of OmniCppComplete, put these in your .vimrc file.

if v:version >= 600
  filetype plugin on
  filetype indent on
else
  filetype on
endif

if v:version >= 700
  set omnifunc=syntaxcomplete#Complete " override built-in C omnicomplete with C++ OmniCppComplete plugin
  let OmniCpp_GlobalScopeSearch   = 1
  let OmniCpp_DisplayMode         = 1
  let OmniCpp_ShowScopeInAbbr     = 0 "do not show namespace in pop-up
  let OmniCpp_ShowPrototypeInAbbr = 1 "show prototype in pop-up
  let OmniCpp_ShowAccess          = 1 "show access in pop-up
  let OmniCpp_SelectFirstItem     = 1 "select first item in pop-up
  set completeopt=menuone,menu,longest
endif

The default key mapping for invoking Omni Completion is C-X C-O. SuperTab plugin can help us invoke Omni Complete when we press TAB key. Also in the default color scheme, vim pop-up menu is in pink. Let’s change that to green.

if version >= 700
  let g:SuperTabDefaultCompletionType = "<C-X><C-O>"
  highlight   clear
  highlight   Pmenu         ctermfg=0 ctermbg=2
  highlight   PmenuSel      ctermfg=0 ctermbg=7
  highlight   PmenuSbar     ctermfg=7 ctermbg=0
  highlight   PmenuThumb    ctermfg=0 ctermbg=7
endif

As we edit code, newer types get defined, but vim continues to use the same tags file. So we need to regularly update our tags file. We can add a shortcut map (say pressing F4) in our .vimrc for the same.

function! UpdateTags()
  execute ":!ctags -R --languages=C++ --c++-kinds=+p --fields=+iaS --extra=+q ./"
  echohl StatusLine | echo "C/C++ tag updated" | echohl None
endfunction
nnoremap <F4> :call UpdateTags()

Vim is all about customizing it to your tastes. You can never get the perfect .vimrc file or plugin collection and settings from anywhere. The trick is to collect cool stuff for your .vimrc and plugins. Vim tips wiki in wikia is a good place for a lot of info. Play around with them. Find what suits best for you. And never forget to share. Happy coding ūüôā

Advertisements

In my previous post, I came up with an enhanced pygenie, which could count total complexity of a Python project as a whole. As explained earlier, my motivation is purely for fun. I took a test run of the enhanced pygenie on some of the popular open-source Python projects. And the results are out ūüôā

Django 1.2.1

time ./pygenie/pygenie.py Django-1.2.1/ -r
Total Cumulative Statistics
----------------------------------
Type         Count      Complexity
----------------------------------
X            1431             1629
C            2393             2408
M            6299            13000
F            1144             3600
T           11267            20637
----------------------------------

real    0m11.840s
user    0m11.489s
sys     0m0.056s

Zope 2.11.4

time ./pygenie/pygenie.py Zope-2.11.4-final/ -r
Total Cumulative Statistics
----------------------------------
Type         Count      Complexity
----------------------------------
X            3005             4549
C            6453             6500
M           23137            45392
F            4985            12960
T           37580            69401
----------------------------------

Totally 5 files failed
FAILED to process file: /home/aufather/Downloads/Zope-2.11.4-final/lib/python/ZEO/scripts/zeoserverlog.py
FAILED to process file: /home/aufather/Downloads/Zope-2.11.4-final/lib/python/zope/app/testing/ztapi.py
FAILED to process file: /home/aufather/Downloads/Zope-2.11.4-final/lib/python/zope/app/component/back35.py
FAILED to process file: /home/aufather/Downloads/Zope-2.11.4-final/lib/python/zope/app/component/site.py
FAILED to process file: /home/aufather/Downloads/Zope-2.11.4-final/lib/python/zope/rdb/gadfly/sqlbind.py

real    0m37.042s
user    0m36.302s
sys     0m0.140s

Twisted 10.1.0

time ./pygenie/pygenie.py Twisted-10.1.0/ -r
Total Cumulative Statistics
----------------------------------
Type         Count      Complexity
----------------------------------
X             935             1282
C            3736             3922
M           16782            26142
F            1323             3268
T           22776            34614
----------------------------------

Totally 6 files failed
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/doc/core/howto/listings/udp/MulticastClient.py
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/doc/core/howto/listings/udp/MulticastServer.py
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/doc/historic/2003/pycon/deferex/deferex-listing0.py
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/doc/historic/2003/pycon/deferex/deferex-listing2.py
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/doc/historic/2003/pycon/deferex/deferex-bad-adding.py
FAILED to process file: /home/aufather/Downloads/Twisted-10.1.0/twisted/python/test/test_win32.py

real    0m20.794s
user    0m20.525s
sys     0m0.036s

Even though I started this activity for fun, I was surprised by the speed and utility of pygenie. Some of the uses I can think of

  • Identify syntax errors within a project. Since pygenie uses the same python compiler, if pygenie could not process a file, the file cannot be processed by Python byte code compiler. Zope has 5 files which will not run while Twisted has 6. Django is clean in this front. When we take a closer look, it is test cases, demo code, tutorials, historical code etc that are broken.
  • Object orientation used within a project. The ratio between methods to functions is a fair indicator. The scores for the three projects are Django(5.5), Zope (4.6) and Twisted (12.7). This is a good indicator of existing style of code.
  • Overall size of a project. Pygenie report indicates that Zope is 3.4 times and Twisted is 1.7 times as big as Django.

The original pygenie is available here. Enhanced pygenie is available from my dropbox here.

Recently I measured some code metrics of a C/C++ project at work. I used C and C++ Code Counter(CCCC) and it is pretty good. There was a similar project at work which was almost completely in Python. I wanted to compare the two and searched for a suitable tool to measure McCabe’s Cyclomatic Complexity(CC) for Python code and found pygenie.

Original pygenie, by default, prints Files(X), Classes(C), Methods(M), Functions(F) only if their CC exceeds 7. If –verbose or -v is given, it prints CC irrespective of whether it crosses the threshold of 7 or not. But I wanted overall CC. My main motivation is purely for fun or for interesting numbers. Kind of like the calories burnt counter on a treadmill or a cycle computer for a bicycle. But overall CC can be useful in estimating testing effort.

Here is a sample run of original pygenie on its own source code.

aufather@simstudio:~/Downloads/pygenie_orig$ ./pygenie.py complexity *.py
File: /home/aufather/Downloads/pygenie_orig/cc.py
This code looks all good!

File: /home/aufather/Downloads/pygenie_orig/pygenie.py
Type Name Complexity
--------------------
F    main 8          

File: /home/aufather/Downloads/pygenie_orig/test_cc.py
This code looks all good!

And here is a sample run of modified pygenie on its own source code.

aufather@simstudio:~/Downloads/pygenie$ ./pygenie.py * -c
File: /home/aufather/Downloads/pygenie/cc.py
This code looks all good!

File: /home/aufather/Downloads/pygenie/pygenie.py
This code looks all good!

File: /home/aufather/Downloads/pygenie/test_cc.py
This code looks all good!

Total Cumulative Statistics
----------------------------------
Type         Count      Complexity
----------------------------------
X               3                4
C               8                8
M              31               70
F              13               26
T              55              108
----------------------------------

I added the total cumulative statistics and a summary for each file. Total is represented by ‘T’. This gives a fair idea of how complex the overall project is. ‘F’ stands for functions, ‘M’ stands for methods, ‘C’ stands for classes and ‘X’ stands for files. Also while I was at it, added a whole bunch of command line options to pygenie.

Here is the help for original pygenie. Valid commands are “complexity” or “all”.

aufather@simstudio:~/Downloads/pygenie_orig$ ./pygenie.py all -h
Usage: ./cc.py command [options] *.py

Options:
 -h, --help     show this help message and exit
 -v, --verbose  print detailed statistics to stdout

And the new options added.

aufather@simstudio:~/Downloads/pygenie$ ./pygenie.py -h
Usage: pygenie.py [options] *.py

Options:
 --version             show program's version number and exit
 -h, --help            show this help message and exit
 -c, --complexity      print complexity details for each file/module
 -t THRESHOLD, --threshold=THRESHOLD
 threshold of complexity to be ignored (default=7)
 -a, --all             print all metrics
 -s, --summary         print cumulative summary for each file/module
 -r, --recurs          process files recursively in a folder
 -d, --debug           print debugging info like file being processed
 -o OUTFILE, --outfile=OUTFILE
 output to OUTFILE (default=stdout)

You can get the full zipped source code from my dropbox here

http://dl.dropbox.com/u/10402332/pygenie.zip.

PS: I based my modification on revision 44 of pygenie. I got the original from http://svn.traceback.org/python/cyclic_complexity/.

PPS: I could not find a better way of sharing the code in wordpress. I have updated the test_cc.py too so that unit test do not fail. If any one knows a better way of sharing code, please let me know. Dropbox works well for me.

Building Gvim from source

Posted: August 15, 2010 in Vim
Tags: , ,

I recently stumbled upon this very interesting vim plugin – OmniCppComplete. It promised to solve one of the problems that has been nagging me. While editing C/C++ code in vim, I have always wished if it could behave like Visual Studio and show structure/class members when we give object. or object->.

I wanted to try it out right away. But now comes the problem. It works only on vim 7 and above. It seems vim 7 onwards, OmniComplete feature is added which allows such stuff. At my workplace, the Debian server where I write code, has vim 6. Tough luck. Plus, I share the server with another 12 users, and do not have root powers. So next option is to build gvim from source. It wasn’t as easy, so this post explains the step I took to get it working.

Before we begin, a brief introduction about building stuff from source in Linux.

Step 0 – Brief introduction to building from source in Linux

Linux being open source, most software for Linux is open source too. That means you have full access to the source code. A bit of googling should get you what you are looking for. Traditionally, make is the utility used to build software for Unix like (and hence Linux) OSes. Linux has its own version of make called GNU make. It should be already installed on you Linux distro. Make depends upon a makefiles which describe what need to be done to build source code. To take care of the variations in Linux distros, GNU came up with¬† Autoconf and Automake which can generate makefiles based upon your particular Linux distro. Typically there will be an executable file “configure” which tells Autoconf/Automake how to generate make files.

So the typical steps to build something from source in Linux is

tar -xzvf source.tgz # untar source code
tar -zjvf source.tar.bz2 # same as above, different compression algo
cd source # move to source code directory
./configure # Autoconf/Automake will generate makefiles
make # make will build the binaries
make install # make install will copy binaries to /usr/bin \
             # or /usr/local/bin (may require root permissions)

Now if you have root permissions, you can follow the above steps. But there are times when you don’t have root permissions, or you do not want to install the binaries in the usual location, we can do this easily while invoking configure. Almost all configure scripts will take “–prefix” as an option. Here you can specify an alternate path where the binaries should be installed to when make install is given.

While compiling, I ran into some issues where configure picked my architecture to be 64 bit. But I wanted a 32 bit application. So I gave “–build” and “–host” options as i386-linux to specify a 32 bit compilation.

pkg-config is a neat utility that can help autoconf in finding the path of the required version of dependent packages. pkg-config will search the path in the same order specified by the environment variable PKG_CONFIG_PATH.

We are going to install gvim and the other stuff into /home/username/usr/bin since we do not have root permissions. So the typical commands that we are going to use are

export PREFIX=$HOME/usr
export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig
export HOST=1686-linux
export BUILD=i686-linux
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install

Also we will keep all source code in /home/username/usr/src. So

mkdir -p ~/usr/src
cd ~/usr/src

Now we are all set. Let’s begin.

Step 1 – Get the source code of course ūüôā

wget ftp://ftp.vim.org/pub/vim/unix/vim-7.2.tar.bz2

The latest version of gvim is 7.3 when I am writing this blog. But I got gvim setup a couple of weeks back, so still have 7.2. There is no separate code for gvim. The code for vim has the code for gvim as well. Gvim depends upon some graphical toolkit like GTK+ (GTK 1.2 from Gnome), GTK2 (current latest GTK from Gnome), Motif (from IBM’s CDE), Athena (from MIT) etc.

Step 2 – Untar it and try compiling

tar -xjvf vim-7.2.tar.bz2
cd vim72
./configure --prefix $HOME/usr --host=i386-linux --build=i386-linux

Wait a minute. Configure could not find any graphical front ends, so skipping gvim. Bad :(. We want to get gvim 7.2 and vim 7.2 is not enough. Typical Linux distributions rarely contain the “devel” packages required for compiling dependent package. Vim could not find the header files of GTK+/GTK2/Motif/Athena (usually found in “devel” package) required to build gvim. Since we do not have root powers, we cannot just do apt-get install.

Step 3 – Get GTK+ and its dependencies’ source code

Well this is not as easy as it sounds. GTK has its own dependencies. I went through the compilation of each source package to list down all the dependencies. Here is the compiled list.

wget http://ftp.gnome.org/pub/gnome/sources/glib/2.24/glib-2.24.0.tar.bz2
wget http://xmlsoft.org/sources/old/libxml2-2.7.1.tar.gz
wget http://download.sourceforge.net/libpng/libpng-1.4.3.tar.gz
wget http://ftp.gnome.org/pub/GNOME/sources/atk/1.30/atk-1.30.0.tar.bz2
wget http://sourceforge.net/projects/freetype/files/freetype2/2.4.2/freetype-2.4.2.tar.bz2/download
wget http://fontconfig.org/release/fontconfig-2.8.0.tar.gz
wget http://cairographics.org/releases/pixman-0.18.2.tar.gz
wget http://cairographics.org/releases/cairo-1.8.10.tar.gz
wget http://ftp.gnome.org/pub/gnome/sources/pango/1.28/pango-1.28.0.tar.gz
wget http://ftp.gnome.org/pub/gnome/sources/gtk+/1.2/gtk+-1.2.10.tar.gz

Since I also use gvim for some coding in Python, I also got Python source code so that gvim was compiled with Python support (+python).

wget http://www.python.org/ftp/python/2.7/Python-2.7.tar.bz2

Step 4 – Untar all the stuff including GTK+ and Python

tar -xjvf glib-2.24.0.tar.bz2
tar -xzvf libxml2-2.7.1.tar.gz
tar -xzvf libpng-1.4.3.tar.gz
tar -xjvf atk-1.30.0.tar.bz2
tar -xjvf freetype-2.4.2.tar.bz2
tar -xzvf fontconfig-2.8.0.tar.gz
tar -xzvf pixman-0.18.2.tar.gz
tar -xzvf cairo-1.8.10.tar.gz
tar -xzvf pango-1.28.0.tar.gz
tar -xzvf gtk+-1.2.10.tar.gz
tar -xjvf Python-2.7.tar.bz2

Notice that we used ‘j’ switch for tar.bz2 archives and ‘z’ switch for tar.gz archives.

Setp 5 – Build them

export PREFIX=$HOME/usr
export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig
export HOST=i686-linux
export BUILD=1686-linux
cd glib-2.24.0
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../libxml2-2.7.1
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../libpng-1.4.3/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../atk-1.30.0/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../freetype-2.4.2/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../fontconfig-2.8.0/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../pixman-0.18.2/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../cairo-1.8.10/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../pango-1.28.0/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../gtk+-1.2.10/
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install
cd ../Python-2.7
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD
make
make install

All the dependencies are met. Now we are all set to build gvim. That leads us to

Step 6 – Build gvim

cd ../vim72
./configure --prefix=$PREFIX --host=$HOST --build=$BUILD \
  --enable-mzschemeinterp --enable-pythoninterp --enable-tclinterp \
  --enable-cscope --enable-fontset \
  --with-compiledby=aufather
make
make install

Ta-da. Gvim is ready. Well almost. If you try to run gvim it will throw this error.

file $PREFIX/bin/gvim
/home/aufather/usr/bin/gvim: symbolic link to `vim'
/home/aufather/usr/bin/gvim --version
/home/aufather/usr/bin/gvim: error while loading shared libraries: \
libgtk-1.2.so.0: cannot open shared object file: \
No such file or directory

Well this is because gvim could not find the shared libraries it requires to run. They are in $PREFIX/lib, but we have to tell that to bash. We do this by setting the LD_LIBRARY_PATH environment variable.

ÔĽŅÔĽŅexport LD_LIBRARY_PATH=$HOME/usr/lib
~/usr/bin/gvim --version
VIM - Vi IMproved 7.2 (2008 Aug 9, compiled Aug 16 2010 01:37:22)
Compiled by aufather
Normal version with GTK GUI.  Features included (+) or not (-):
-arabic +autocmd +balloon_eval +browse +builtin_terms +byte_offset
+cindent +clientserver +clipboard +cmdline_compl +cmdline_hist
+cmdline_info +comments +cryptv +cscope +cursorshape +dialog_con_gui
+diff +digraphs +dnd -ebcdic -emacs_tags +eval +ex_extra +extra_search
-farsi +file_in_path +find_in_path +float +folding -footer +fork()
-gettext -hangul_input -iconv +insert_expand +jumplist -keymap
-langmap +libcall +linebreak +lispindent +listcmds +localmap +menu
+mksession +modify_fname +mouse +mouseshape -mouse_dec -mouse_gpm
-mouse_jsbterm -mouse_netterm -mouse_sysmouse +mouse_xterm -multi_byte
+multi_lang -mzscheme +netbeans_intg -osfiletype +path_extra -perl
+postscript +printer -profile +python +quickfix +reltime -rightleft
-ruby +scrollbind +signs +smartindent -sniff +statusline -sun_workshop
+syntax +tag_binary +tag_old_static -tag_any_white +tcl +terminfo
+termresponse +textobjects +title +toolbar +user_commands +vertsplit
+virtualedit +visual +visualextra +viminfo +vreplace +wildignore
+wildmenu +windows +writebackup +X11 +xfontset +xim
+xsmp_interact +xterm_clipboard -xterm_save

You can put these lines in your ~/.bashrc

export LD_LIBRARY_PATH=/home/username/usr/lib
alias vim='/home/username/usr/bin/vim'
alias gvim='/home/username/usr/bin/gvim'

That’s it. vim and gvim 7.2 is ready on your machine. Take it out for a spin ūüôā