FreeBSD review and howtos from a Linux user

3 03 2008

I recently decided to give the new 7.0 release of FreeBSD a go and was fairly impressed. I did use BSD along time ago on a home server for a few months but pretty much forgot everything about it from back then.

FreeBSD Screenshot

Introduction
Firstly FreeBSD refers to both a kernel and userspace tools making it a whole operating system (userspace tools being the basic programs like shells and copy/move commands), this is different to Linux which is just a kernel and distros are technically called GNU/Linux to show that it is using the GNU userspace tools. You can install the GNU userspace tools on FreeBSD and you can also get GNU/FreeBSD hybrids such as the Debian GNU/kFreeBSD, and there was work on a Gentoo/FreeBSD but it never went anywhere, although there not really used much. Theres also not a huge point in either since all the userspace tools are based on the original Unix ones and try to mee POSIX standards etc %90 of the functionality is the same. There are some differences, for instance ‘ls –color’ on FreeBSD is ‘ls -G’, some commands require the flags to be in the correct order so ‘cp /directory /somewhere -rf’ won’t work as -rf is at the end instead of the start befoure the directories, when hitting the down arrow at the end of a man page it will exit on FreeBSD.

The other important difference between BSD’s and Linux is the license they use, Linux uses the GPL and BSD uses the BSD license. The GPL is actually more restrictive but in a way that is designed to guarantee everyones overall freedoms, by enforcing that the source remains open when redistributed. The BSD license basically says you can do whatever you want provided you keep the credits in, including taking the source and closing it or relicensing it. Apple took the FreeBSD userspace tools for OSX (and the Mach microkernel)

The difference in the kernels means that hardware is going to behave differently, device names in /dev/ are diffrent for starters. Ethernet cards have names that match the device model, for instance instead of “/dev/eth0” you might have “/dev/re0”. My harddrive is “/dev/ad10s1” instead of “/dev/sda1” There will be differences in the hardware that is supported, wifi is apparently better on FreeBSD although there has been a lot of work in the Linux area recently and some of the drivers have been ported across (leading to some controversy due to the relicensing of BSD code under the GPL, although it has apparently been resolved).

The overall system feels fairly similar to Slackware and Gentoo, except with better package management IMHO.

Installing
Installing isn’t particularly hard, if you have installed either, its not a Ubuntu style userfriendly distro so you do need to partition and such but it didn’t require heaps of planning and was fairly hassle free. There is also the handy FreeBSD handbook which is actually fairly good, normally when I get told to read opensource documentation I find a bunch of out of date or setup in a completely different way by everyone but the guide writers. You can get away with only getting the 1st CD since that contains the basic system and if you don’t choose much in the way of packages you won’t need the others. If however you do get asked to insert the 2nd or 3rd CDs, at that stage you can just ctrl+c and reboot into a working distro and use ‘sysinstall’ to make any additional changes. Set the root password with ‘passwd’ and add a user account with ‘adduser’.

One installed you need to setup stuff, once again if your a Gentoo user this is familiar. FreeBSD has binary packages that can be retrieved automatically via ftp by using the ‘pkg_add -r ‘ command. FreeBSD however also can compile packages from source using ports, this is similar to Gentoo’s portage which was heavily inspired by ports. Ports seems a simpler and more stable system than portage with which I have had problems maintaining when updating packages can break or lock other packages.

Firstly I had some problems with my USB keyboard, in the end I had to disable USB legacy in the BIOS which was blocking BSD from using it for some reason, this has the unfortunate side effect of disabling the keyboard in the boot loader but its not needed for now.

Ports
Ports is the system that FreeBSD uses to get source code for packages, compile and install it. It is similar to Portage as it used ports as inspiration. Compiling from source has advantages over using binary packages. For instance patent issues aren’t a problem because its the source code, I am not a lawyer but source code itself cannot be patented as its a blueprint not an actual thing, and compiling the code your self is possibly ok since you can apparently violate patents for personal use (not too sure on that, some talk here, at the very least it means that to enfoce the patent requires going after individuals rather than FreeBSD so home users are unlikly to be at risk (unless patent lawyers frequent your premises), it also allows for people in countries without such laws access to the code), but as I said I’m not a lawyer and haven’t bothered to find much in the way sources. Secondly you get more control over what features you want in your program and thirdly you can optimize the program for a specific architecture.

Setting up ports, If you want to grab a new snapshot of ports you will need to run ‘portsnap fetch’ followed by ‘portsnap extract’. If you already have “/usr/ports” perhapses because you chose it when the installer asked you can apparently ‘cd /usr/ports’ followed by ‘make update’. This should use portsnap to update it. The full was about a 50mb download when I did it with ‘portsnap fetch’.

Ports has a similar thing to Gentoo’s portage useflags called KNOBS, you can see a whole list of them in /usr/ports/KNOBS. I believe you can also set CFLAGS like in Gentoo although I havn’t bothered.

Desktop
If your using it on the desktop you will need to install xorg and a desktop environment, you can “pkg_add -r gnome2” to do that, which should pull in all the dependencies. Alternatively to build from source: cd /usr/ports/x11/gnome2 && make install clean
If you want to use gdm add gdm_enable=”YES” to /etc/rc.conf. If you don’t want Gnome you can so whatever other environment you want, Fluxbox, XFCE, KDE, etc… You will need to edit ~/.xinitrc and add “gnome-session”, “fluxbox” or whatever. You can then ‘startx’. At this point there will be a bunch of programs you need to install either from ports or with pkg_add -r such as firefox, pidgin, vlc (although totem should be in if you did gnome2), music players, etc…

Shell
The default shell csh might not be to your liking, a lot of people are experienced with bash which is the choice of most Linux distros although personally I prefer zsh which has better tab completion and fancy prompts included. ‘pkg_add -r zsh’ or ‘pkg_add -r bash’ or install them from ports. Then run ‘chsh’ which will give you a vi window, change the “Shell: /usr/bin/csh” to “Shell: /usr/local/bin/zsh”. Bash users will need to change it to bash obviously. If vi is a problem for you, you can probally edit ‘/etc/passwd’ with whatever you want. You will need to setup your shell rc, ~/.zshrc for zsh (although it has a config window on first time login).

Sound
You will probably need to enable sound, Theres a handy howto here, but basically ‘kldload snd_driver’, ‘cat /dev/sndstat’, look for the driver name right after “kld”, add DRIVERNAME_load=”YES” to /boot/loader.conf

Mount Linux ext3 Drive
Mounting your Linux drive, add the following entry to /etc/fstab: /dev/ad10s2 /mnt/ubuntu ext2fs rw 0 0 with whatever changes you need for your device name or mount point. You will need to make the mount point to with ‘mkdir /mnt/ubuntu’. If you get the error “mount: /dev/ad10s2 : Invalid argument” this could be because your drive is unclean, you will need to install the /usr/ports/sysutils/e2fsprogs/ and run ‘fsck.ext3 /dev/ad10s2’ (Assuming your using ext3).

NTFS read/write with NTFS-3g
Mounting a NTFS drive with read/write is a bit more work, you need to install ‘/usr/ports/sysutils/fusefs-ntfs’, however it needs the userspace source. You can get this with ‘sysinstall’ choosing “Configure>Distributions>src>All”, this will grab all the source its probably not all required but I couldn’t find the specific package and its handy to have the rest around anyway. Once you have that you can ‘ntfs-3g /dev/ad10s1 /mnt/windows’. Adding an fstab entry is again a bit problematic. Under 7.0 the mount command has had hardlinks to what it can call, mount_ntfs-3g isn’t in that list so you either need to patch mount or rename the current mount_ntfs, and link ntfs-3g in its place. Without that this was throwing an “mount: /dev/ad10s1 : Operation not supported by device” error.

nVidia drivers
The drivers work quite well on FreeBSD 7.0 but only on i386 (64bit is out of luck thanks to closed source nvidia drivers, 64bits OS is mostly useless anyway), the official ones refuse to install for me but the ones in ports work. Firstly you need the kernel source code installed, if you followed the NTFS-3g instructions above you will have the source already. If not you can specify just the base and sys source in sysinstall which is all thats needed for the drivers. Then cd /usr/ports/x11/nvidia-driver && make install clean Add this to /boot/loader.conf nvidia_load="YES" Now you need an xorg.conf file, by default BSD just relies on the autoconfig magic in the latest xorg releases so we need to force it to generate one with: Xorg -configure then copy the newly created xorg.conf.new to /etc/X11/xorg.conf. If you don’t want to edit the xorg by hand cd /usr/ports/x11/nvidia-xconfig && make install clean then nvidia-xconfig will change the “nv” driver to “nvidia”.

Compiz Fusion
Firstly if your on nVidia hardware you need to enable some xorg.conf settings, you can do them with nvidia-xconfig if you installed it above nvidia-xconfig --composite && nvidia-xconfig --render-accel && nvidia-xconfig --add-argb-glx-visuals -d 24

Now install Compiz with: cd /usr/ports/x11-wm/compiz-fusion && make install clean When you want to start it use LIBGL_ALWAYS_INDIRECT=1 compiz --sm-disable --replace ccp and for the Emerald decorationsemerald --replace. I had some problems with emerald not starting right away and drawing no decorations but after a while it loaded fine without me noticing and now starts fine. You will probably want to make Compiz start automatically, there are various guides on Compiz around that give you different ways. The config programs should be in System>Preferences for Compiz and Emerald, otherwise use ‘ccsm’ and ’emerald-theme-manager’.

updatedb
For those of you who use ‘locate’ command to find files, you will probably want to update the database, you might have already noticed that ‘updatedb’ isn’t a command. The actual command is ‘/usr/libexec/locate.updatedb’ but this will give you a warning as running it as root will expose hidden files that only root or specific users should see to the database (not such a problem for single users), you can however run the cron job manually for a safer database with: /etc/periodic/weekly/310.locate

Flash
One of the main problems with Desktop FreeBSD is that Flash has no native port for FreeBSD thanks to Adobe (Even though there is a Solaris port thanks to Sun working with Adobe, you can have your say on the flash development blog, also an online petition exists for what little use they are with over 5000 signatures(and spam bots)). There has been some talk on the mailing lists of a native FreeBSD port.

The only real solutions seems to be to use Flash 7 or possibly Windows Firefox under Wine ☹ (PC-BSD Apparently ship a Windows Firefox PBI)

If you want to try the Linux version of Flash 9 which freezes for me (and everyone apparently), you need to install the Linux plugin /usr/ports/www/linux-flashplugin9 and the wrapper /usr/ports/www/nspluginwrapper/. Then run nspluginwrapper -v -a -i to register the plugin. Check in ‘about:plugins’. Alternatively you can try Flash7 with a wrapper, there is also another wraooer /usr/ports/www/linuxpluginwrapper but it wouldn’t install for me.

ETQW [unsolved]
I though I would give ETQW a go under FreeBSD, unfortunately I ran into problems with the Linux compatibility layer, I’ll post the steps here in case its of use to someone trying to do something similar or can offer a solution.

Firstly the installer didn’t give the default path but entering one works however it then failed to make the path so you need to make it manually, then it failed to read from the cdrom (although I didn’t really look into it since you can just copy the files across manually, possibly manually mounting it manually would have solved it). Copy the correct files into the base directory.

When running the etqw binary I get an error about missing sdl libraries “./etqw.x86: error while loading shared libraries: libSDL-1.2.so.0: cannot open shared object file: No such file or directory”, You can fix this with cd /usr/ports/devel/linux-sdl12/ && make install clean

You will need to brand you etqw binaries as Linux ones, change to your etqw directory and run brandelf -t linux * also brandelf -t linux pb/* might be needed.

Then run: portmaster emulators/linux_base-fc4 to get all the Linux compatibility stuff installed (if using portmaster you need /usr/ports/ports-mgmt/portmaster/ installed)

Then there I got “./etqw.x86: error while loading shared libraries: /usr/local/lib/libX11.so.6: ELF file OS ABI invalid”, make sure that /usr/ports/x11/linux-xorg-libs/ is installed.

For the error “./etqw.x86: error while loading shared libraries: libjpeg.so.62: ELF file OS ABI invalid”, make sure you have “/usr/ports/graphics/linux-jpeg”. You can also try ‘sysctl kern.fallback_elf_brand=3’. But those didn’t work for me, I notice that etqw ships its own libjpeg, I tried replacing that with the /usr/compat/linux/usr/lib/libjpeg.so.62 one it seems to work but now I get another error “./etqw.x86: error while loading shared libraries: /usr/local/lib/libX11.so.6: ELF file OS ABI invalid”, possible its loading the ones in /usr/local/lib instead of /usr/compat/linux/usr/X11R6/lib/libX11.so.6 but I don’t know and thats about all I can think of now.

Also make sure you have linux_enable=”YES” in /etc/rc.conf and run ‘/etc/rc.d/abi restart’ to enable the compatibility layer stuff if you didn’t already have it done.

Wine
Wine works fine under FreeBSD, to install: cd /usr/ports/emulators/wine && make install clean

Mounting samba shares
Firstly you need to install Samba with /usr/ports/net/samba3/ && make install clean then you can manually mount it with the following: mount_smbfs //username@host/share /mnt/mointpoint

the -o options that you use in Linux such as username, password, credentials don’t work on FreeBSD’s version. If you want to make it automated, edit /etc/fstab and add: //username@server/share /mnt/mountpoint smbfs rw 0 0 Then edit /etc/nsmb.conf and check the example near the end. Alternatively you can put them in ~/.nsmb.conf for a per user solution.

Advertisements




code, code.back, code.back2… – A better way with Revison Control (svn/git/bzr/hg tutorials & comparisons)

18 06 2007

This article is very long, it covers some basics of what revision control system (RCS)/ source code management systems (SCMS) are, basic tutorial of using subversion for a personal repository, what distributed ones are, basics of using git, bzr and hg for a personal repository and my comparisons on them. Its only a basic introduction, I’ve never had to manage any large complex projects so advanced stuff isn’t covered (plus its long enough).

If you program and don’t use some kind of rcs you are making your life much harder than it should be, rcs are a great, distributed ones are greatest. All you need is to learn a few steps to setup a repo, and somewhere to put it, anything with ssh can be used or just on the local disk.

Even for non-programmers, if you find yourself making changes to config files much then having a repository containing them is definitely a good idea, if you botch it up, you can always revert to the previous edit and compare the 2 with diff.

    Introduction to RCS


Originally when I would code, I would intermittently ‘cp -rf directory directory.backup’, that way if I screwed up my code I could always go back. This was working fine for my smaller projects, at least until one particularly painful Uni assignment (SunRPC will segfault on anything), eventually I had reached backup.22 and often I had to go back a few revisions, Not an easy task because I wouldn’t remember the exact number, and I had often done more than one change to the code like add comments to everything, which resulted in me creating more backups with things like a single comment added, because the code I had done had started to randomly segfault. I’m sure there was a simple memory leak but with the deadline a few hours away I didn’t have time to hunt it down (basic gdb wasn’t working because it was the SunRPC libs that where crashing). In the end I got my assignment in (although it was probably the worst mark on an assignment yet, once again I hate RPC).

After that I decided to try using a revision control system, previously I had never actually though about using them for my simple coding and just assumed they where only needed for larger project, the only time I had encountered them was to ocassionaly grab some code from when I needed something newer than was shipping with my Linux distribution, however while googling for stuff about uni I managed to find this website from another student about setting up SVN for projects. I had only previously used svn for grabbing code from public projects, I had also used cvs although it was fairly clear than cvs was a fairly outdated system.

    SVN – Subversion tutoiral


SVN works rather well for me as it is on the systems at uni and can be tunneled over ssh so I can push/pull to/from my server at home. The basic functionality of svn allows for going back to any previous revision with ‘-r #’, coding from any system that can connect to the repository all I need to do is checkout/update it, seeing a ‘diff’ between revisions to see what I changed.

Unfortunately subversion isn’t distributed (explained later) so I wouldn’t recommend it, but understanding the basics of revision control is important, so I have instructions on using it here, the same basic outline of commands is used for most of the revision control systems around with a few minor differences. I might use svn for basic repository for editing config files but any of the distributed ones would work just as well.

You can use any system you either have direct access to or ssh (and http etc…) to store your code, I’m using ssh in this.

SourceForge (from the owners of Slashdot) provide free public svn (and cvs) hosting for open source projects, include bug tracking and basic forums however I haven’t found the site very nice to navigate, although you can just host a normal website on it.

Setting up a personal local repository is easy.

Firstly we need to make a svn folder where all the other svn projects will live:
mkdir ~/svn

Then we need to make a repository for the project:
svnadmin create ~/svn/PROJECTNAME

Next is importing the current code, you do this from the directory where your code lives not the svn created one, make sure you clean up any unneeded files like binaries and generated output first:
svn import . \
file:///home/USERNAME/svn/projectname/trunk -m "Initial commit."

Notice that it is going into the sub folder trunk, this is important because later on you might need to tag code so you might end up with /trunk/, /1.0rc1/ and /1.0/, you can just put the code in the main directory if you don’t want this kind of functionality. Make sure there are 3 /’s in the uri, normally the server name goes after the first / but since this is local there aren’t any. You must also specify the full path to your folder. -m is the commit message that describes the changes for revisions.

You can also use svn+ssh://USERNAME@SERVER/home/USERNAME/svn/PROJECTNAME/trunk if you want to do it over ssh.

The next set is to checkout your repository, even though you have a local copy you still need the subversion metadata (Annoying url prefix, I wish it was just ssh://):
svn checkout \
svn+ssh://USERNAME@SERVER/home/USERNAME/svn/PROJECTNAME/trunk \
PROJECTNAME

This time I’m doing it over ssh, once again remember that its coming from the trunk folder. The trailing PROJECTNAME is to make svn rename. co is a shorter alias of checkout if your excessively lazy.

Thats the hard bits done, from now on its very simple as all the information about where to upload is stored in the .svn folder in your project.
Now you just edit your code, and once your happy with the changes you type:
svn commit -m "Description of changes."

When you create a new file that you want to add to the repository you must first tell svn that you want to add it manually, this avoids accidentally uploading compiled binary files or files outputted by your program:
svn add filename

To update to the version of the code in the repository (or a particular version with -r#):
svn update

To see the difference between revisions, you can also specify a particular revision with -r:
svn diff

To see the logs:
svn log

To make a tag:
svn copy \
svn+ssh://USERNAME@SERVER/home/USERNAME/svn/PROJECTNAME/trunk \ svn+ssh://USERNAME@SERVER/home/USERNAME/svn/PROJECTNAME/1.0

    Distributed Revision Control Systems


SVN was a massive improvement to managing even simple personal code, I used it for several months without issues, however there is a new bread of RCS that are appearing, distributed ones. There are currently 3 main contenders:

  • Git – Made by Linus for maintaining the Linux kernel, also used by KDE.
  • Mercurial – Run with the command ‘hg’, A popular Python based one, used by Sun (for hosting of Java).
  • Bazaar-NG – Or ‘bzr’ Python again, as used on launchpad.net and the Ubuntu community (there all Canonical made).
  • There is also darcs (written in Haskell), GNU Arch, and monotone. There is a Wikipedia article listing various revision control software (commerical/free and central/distributed).

    Being distributed means that when you check out a central repository, you actually have your own local repository rather than just a copy of the code from it, so you can commit changes without having access to the central repository. Allows for much easier experimentation as you can quickly branch off from your local repository and its Useful for people with laptops who might not have an internet connection. With subversion, you can checkout a repository but then your stuck with the one version, you can only commit back to the main repository, the most you could do is try to copy the directory and other painful workarounds. Also there isn’t technically a ‘central’ repository, although there will generally an official one everyone downloads from. Still handy features to have even when its just for personal use, for instance a simple ‘svn log’ needs to talk to the central server, which can take some time if its a large repo and/or is over a slow connection.

    Speed wise Git is currently the fastest for most operations as it was designed for maintaining the massive Linux kernel. Next fastest is Mercurial and then Bazaar (which is planning to match git speeds in its 1.0 release). However for most simple projects speed isn’t that much of a requirement, as long as its not tediously slow for simple changes any of them should work fine.

    The functionality of all of these are fairly similar, you tell it who you are, you init the original source directory, commit the initial repository, then you can checkout from anywhere with access, branch off code, modify code, commit it, merge it back into the master branch, push it to the server. Review Logs, see changes with diffs etc…

    Most of these support the ability of checking out code from repositories of a different type, you might need a plugin though. You can also convert between systems with tailor, although you might loose some information.

    In the end its probably just personal choice which one you prefer as they all offer the same basic functionality.

      DRCS Tutorials


      Git


    Firstly there is a great talk from Linus about Git on Google video, its 1hr 10min long. It might be somewhat dated however, some of the functionality talked about might have been implemented or speedup since then (for instance pushing in git now exists).

    Git is written in c and is currently the fastest. It is probably best suited for larger projects. However some of Git is more advanced features are a bit harder to use and understand although not by too much for basic usage, so it might not be suited for the less experienced user. The speed improvements on Git are apparently lost on Windows systems as they rely on some specific methods of disk access (unless this has been fixed in newer versions). So Windows or multisystem developers might want to avoid it.

    If you want, you can get free public Git hosting here, although its only a very basic service currently.
    UPDATE: There is also github which has a free opensource developer plan (100mb, no private repos).

    A nice thing about Git is that it keeps all your branches in the same folder, with bzr/hg when you branch of it creates a separate folder for that branch, you could keep them all in one main project folder (For bazaar you can create a repository that stores all your branches saving space by sharing common files) but with Git everything is in the one folder by default making for a much tidier feel, branches you aren’t working on are tucked away and you switch between them fairly painlessly with the checkout command. Might require a bit more effort to work between 2 branches however.

    Git also has nice sha-1 ids for everything so you can tell if things become corrupt, and it generally views all your code as one thing rather than each file so it can track changes to a function even if its moved from one file to another.

    You can ‘apt-get install git-core’ on Ubuntu/Debian, however its out of date so the instructions will vary. You can get the code from the site compile from source for a newer version.

    Firstly tell Git who you are (and enable prwdy colours), the following for newer version of Git:
    git config --global user.name "YOURNAME"
    git config --global user.email EMAIL@DOMAIN.com
    git config --global color.diff auto
    git config --global color.status auto
    git config --global color.branch auto

    Note that those are all –config, not -confg, wordpress screws it up

    To initialize the current code directory (older versions use ‘git-init-db’):
    git init

    When committing to Git, you need to maintain an index of files that are to be committed, you can use the ‘add’ command to do this, in svn you only need to add new files to but in git you need to also add changed files, however rather than adding changed files manually you can use ‘commit -a’ which will automatically add the changed files to the index (but not newly created ones). Since all your files are new in this initial import you need to add them:
    git add .

    Then commit them:
    git commit

    When you want to grab your code from a remote repository and put in in the current directory, use:
    git clone ssh://SERVER/home/USERNAME/git/PROJECTNAME

    Enter your directory, you can then make a branch for hacking on:
    git branch BRANCHNAME

    View your list of branches:
    git branch

    Then you switch to that branch:
    git checkout BRANCHNAME

    Modify some code and check it into your local BRANCHNAME branch:
    git commit -a

    Switch back to your original local branch:
    git checkout master

    Merge the changes into the master branch:
    git merge BRANCHNAME

    Delete the extra branch (-D will force it to delete if you didn’t merge it):
    git branch -d BRANCHNAME

    Push the branch to your server:
    git push ssh://USERNAME@SERVER/home/USERNAME/git/PROJECTNAME

    Theres some more tutorial information on Git here.

      Bazaar – (bzr)


    Bazaar written in python is probably the slowest of the 3, however the current project roadmap for 1.0 is to match the speed of git, so there might be some improvements appearing. There are benchmarks here showing much better speed improvements, up to 0.15, no 0.16/0.17 which also list more performance improvements in their changelogs. I haven’t found any videos on Bazaar but there have been three, shuttleworth, posts recently on bazaar as a lossless RCS.

    For public Bazaar hosting there is launchpad, which has bug tracking and such for project, and storing personal user branches.

    Bazaar seems fairly simple to use, I haven’t needed any of the more advanced features but it seems like advanced stuff would be simpler under Bazaar than Git, but for the simple stuff there isn’t any major difference.

    Firstly set your name:
    bzr whoami "Your Name <EMAIL@DOMAIN.com>"

    Enter your source code directory and initialize it:
    bzr init

    Add the files to the index:
    bzr add .

    Commit the branch. this same command is also used to commit code after its modified, by default it will add all changed files to the index, like -a in git:
    bzr commit

    You can create a repository to store branches, this allows you to save space by sharing the common files between them.
    bzr init-repo REPONAME
    cd REPONAME

    Now you can branch off from your remote branch into the local repository, notice its sftp for ssh now, a different standard for the same thing again, you can use ~ for the home folder now though, there is also bzr+ssh:// which doesn’t seem to need the paramiko library but i’m not sure of the difference between them other than that:
    bzr clone sftp://USERNAME@SERVER/~/bzr/PROJECTNAME

    In addition to ‘clone’, you can also use ‘checkout’, this means that any changes you commit, as well as being committed to the local branch will also be committed to the branch you checkout from, if possible. This is somewhat similar to svn, except changes are still committed to the local branch regardless of the remote branch being accessible (unless you use –lightweight, in which case it works just like svn and all everything depends on the remote branch working). You can also use checkout inside a branch to obtain the latest committed version of that branch into the working directory which is sometimes needed if you push branches as it will transfer the .bzr directory with the revisions but not the working branch.

    You can fork of from your local branch for experimental coding, which will make a separate folder in the repository:
    bzr clone PROJECTNAME PROJECTNAME-testcode

    Then after coding, change to the main local branch directory and merge:
    bzr merge ../PROJECTNAME-testcode

    Then you can push the local branch back to your servers branch:
    bzr push sftp://USERNAME@SERVER/~/bzr/PROJECTNAME

    Also see the official Bazaar tutorial.

      Mercurial – (hg)


    Mercurial works basically the same as bazaar. Theres a google video tech talk on it here (50min).

    Thus you must firstly identify thyself:
    echo -e "[ui]\nusername = YOUR NAME <EMAIL@DOMAIN.com>" > ~/.hgrc

    Changeth to thy source code directory and initilizeth with:
    hg init

    Addeth ye new files to thy index:
    hg add

    Commiteth thy files to thy repo:
    hg commit

    Snag your remote repo to a local location:
    hg clone ssh://USERNAME@SERVER/~/hg/PROJECTNAME

    Branch off your local main to a secondary branch:
    hg clone PROJECTNAME PROJECTNAME-testcode

    Modify some code, and commit to the secondary branch with:
    hg commit

    Change back to your primary local branch and merge (this needs 2 commands):
    hg pull ../PROJECTNAME-testcode
    hg update

    Push it to your remote repo:
    hg push ssh://USERNAME@SERVER/~/hg/PROJECTNAME

    Official Mercurial Tutorial.

      Finally


    After trying out all 3, I found them to be vary similar to each other and any would be suitable for most purposes, you could probally pick one at random and be happy or choose one based on the public services that are avilable such as launchpad, I will probably end up using bzr, hg seemed to make merging a bit more of a pain, requiring an extra step, and the ‘merge’ command some how changed from the docs to the ‘update’ command, also the aesthetics of the output wasn’t as good but thats a bit nitpicky. Bazaars rapidly improving speed should see it ahead of hg if they meet their goals. I also liked git quite alot and might use that for some stuff but it isn’t available on the Solaris systems at uni, and requires 22mb just for the basic binaries so to much for me to install locally (50mb directory limit), but I do favor the approach of having all the branches in the one local location rather than making a whole new one each time, cuts down on the appearance of clutter.

    If you are looking for public hosting for your code with a repository of your choice, you can check this wikipedia article which shows a handy list of hosts and what systems they support.





    SunRPC beginner tips

    13 06 2007

    RPC (Remote Procedure Calls) are used for making client/server programs where the client can call a function on the server without having to implement their own network code, they link in with rpcbind which is run on just about every system now days. It works by using a simple msg.x template, running it through a code generator rpcgen and linking the results with the client and server code.

    I had to do 2 assignments this year for my distributed computing class using Sun’s RPC on Solaris.

    RPC is extremely painful and there isn’t to much in the way of beginners resources. The server kept core dumping on me and the bit causing the problem was often in the RPC libs rather than my code. Normally this is caused by breaking memory allocation but its very hard to track down, gdb doesn’t really help.

    A few tips i picked up while working on Sun RPC:

  • You can stop the server for backgrounding, allowing you to use standard printf’s for debugging. Just add -DRPC_SVC_FG to your servers compile line.
  • Correct memory management is crucial, any leaks no matter how minor will cause crashes with RPC.
  • Don’t leave any pointers undefined even if you haven’t put anything into them and are using a size variable of 0 (such as emptry arrays/strings), NULL them, SunRPC will try to free() any pointers after receiving the struct back which can cause your client to crash when it recives a successful response because its trying to free the memory before putting the actual response struct data in there, since they are undefined it will try to free random memory and segfault.
  • Make sure this applies to the both the sent and result struct, the server should automatically initialize the result struct as you can’t guarantee the client has passed a valid empty struct
  • Make sure you test more than one remote function in a row, quite a few memory errors will not manifest until the server attempts to process the next request AFTER the bad function has worked. If your server is seg faulting on a function, make sure its actually not the previous function that is the problem.
  • Its much easier to use one universal message struct for all the remote functions to be passed back and forward rather than making a new struct type for each function. Possibly slightly less efficient but not by too much for simple projects.
  • Its much easier to let the server do all the work, if the information is just getting printed to the console passback a string rather than a struct with the information in it. Probably not a good practice for real life usage though, but much eaiser for learning basic RPC.
  • The Linux rpcgen seems to be fairly horrible, most of my code wouldn’t work with it, doesn’t seem to support generating stubs, maybe there is an entirely different approach for programming rpc in Linux but I couldn’t find it. Current version might be broken. Things like enums just wouldn’t work for me (which awas a problem because my 1st assignment specified the them). Some sample code I downloaded worked fine, others just wouldn’t.
  • You can make your code thread safe and handling concurrent connections with ‘rpcgen -MA’, you will still need semaphores or some other form of concurrency control.
  • Sun’s rpcgen has the ability to generate template stubs for server and client code with -a, very useful, they are called msg_server.c and msg_client.c, done actually modify them as they will be overridden next time, just copy em. Can be used in combination with the -aAM for threadsafe.
  • Code generated by rpcgen is outdated and will give warning when compiled but still works on, you can fix the stubs by adding int befoure main and such but the templates you will probably need to live with.
  • Variable sized arrays in the template file are declared with not [], you can specify a max length ie , [255] will do fixed size arrays like normal.
  • RPC has a ‘string’ variable type in the template file, this is the equivalent of c’s char* (notice its string, not c++’s String) for example: ‘string name;’, your c code will see this as a normal char*
  • A array in the RPC template file will make a struct with name.name_val and name.name_len
  • char* for strings in the template file was causing me pain, I don’t remember why but there is probally a reason for string.
  • Remember to NULL terminate your strings when they are passed around, otherwise RPC won’t know when to stop freeing
  • Functions can only accept one struct, so make sure it contains everything needed.
  • Sometimes poking into the files generated by the template can help understanding some problems, such as typos in msg.x not being caught by rpcgen but causing your source to fail compiling.
  • If possible use something other than RPC (CORBA, XMLRPC, SOAP), I haven’t used them but they can’t be worse. They might have some overhead though.
  • Linkage:
    http://www.cdk3.net/rmi/Ed2/SunRPC.pdf – SunRPC definition.
    http://www.cs.cf.ac.uk/Dave/C/node34.html – Some basic rpc examples, the way it works is a bit different to the stubs generated by Sun’s rpcgen but its fairly easy to work out the changes needed.
    http://www2.cs.uregina.ca/~hamilton/courses/430/notes/rpc.html – Same again, includes a linked list example.





    Linux/UNIX Permissions

    17 04 2007

    I’m writing this because people sometimes seems to have trouble understanding the permissions under unix and get confused with setting permissions like 777 or 555.

    Permissions
    Basically every file and directory has a username and group associated with it. It then has 3 sets of permissions for owner and group and other.

    root user and the owner of the file have full control as to what permissions can be set, owner can for instance can remove roots permissions to read it but this can be overridden by root. But it is important to remember that by default root would be blocked for accessing the file until it is overridden so things like auto cleaning scripts to remove files from a directory could be bypassed if they aren’t coded to correctly ignore permissions, this can be an advantage if you want to allow users to be able do so.

    The 3 sets of permissions each consist of 3 settings, Read, Write and eXecute which being binary are either on or off.

    For files what each of these does is fairly obvious:

  • Read allows you to read data from the file
  • Write allows to the modify the file (including deleting it)
  • execute chooses weather you can run the file, the execute bit isn’t secure as it would be possible to use another program to execute the file regardless of whether the execute bit is set such as using sh to call a shell script directly so you would need a fairly heavily locked down system until you can be %100 sure that a file with the execute bit disable won’t be executable by someone who is going out of their way to do so. You also need to be able to read a file in order to execute it.
  • Extra info: Execute can also be set to be S instead of x, this allows the executed program to be run with the permissions of the owner of the program, rather than the permissions of the user running it. This can be a very bit security rist.
  • Directories are a little bit different:

  • the execute bit decides weather you can enter the directory so you can’t ‘cd /directory’ into the directory but you can ‘ls /directoy’.
  • Read is used to determine if you can list the contents of the directory so you can block the ability to use ls to list the contents but still allow a user to enter the directory with cd by allowing execute, is it is possible to create a file in a directory that can be accessed by specifying the full path name without being able to browse the directory itself
  • Write allows you to create files in a directory, and also delete/rename the directory itself and files inside the directory (regardless of owner). You can write to a file without having read access
  • Extra into: Write access leads to a problem, user can delete/rename the directory itself or files that aren’t theirs, if you want a directory that users can create files in but you want to stop them from deleting it, such as /tmp this is solved by having an extra bit called the sticky bit (+t), t only shows up for the ‘other’ user since the owner and root are expected to beable to delete their own directory. If the /tmp is missing the sticky bit then a user can cause havock with the system by deleting the tmp directory that is require for a lot of programs. Files can also have the sticky bit but it is ignored nowdays, it was designed to allow the files to ‘stick’ in memory.

    STICKY FILES
    On older Unix systems, the sticky bit caused executable files to be
    hoarded in swap space. This feature is not useful on modern VM sys‐
    tems, and the Linux kernel ignores the sticky bit on files. Other ker‐
    nels may use the sticky bit on files for system-defined purposes. On
    some systems, only the superuser can set the sticky bit on files.

    STICKY DIRECTORIES
    When the sticky bit is set on a directory, files in that directory may
    be unlinked or renamed only by the directory owner as well as by root
    or the file owner. Without the sticky bit, anyone able to write to the
    directory can delete or rename files. The sticky bit is commonly found
    on directories, such as /tmp, that are world-writable.

  • Permissions as shown in a ls:
    U G O ref user grp size date time name
    drwxrwxrwt 13 root root 16K 2007-04-17 23:09 tmp

    The ‘d’ stands for directory, for normal files this will be a ‘-‘. Next we have the 3 permissions for owning user (root) ‘rwx’, then the 3 for group (root) ‘rwx’ and then the 3 for other users ‘rwt’. So root and owner have full permissions (in this case the owner is also root) but all other users have almost full permissions but cannot modify the directory itself.

    Then there is a counter, this isn’t important to under stand by it counts how many times that file/directory is referenced when it is 0 the file system will consider that space to be free space and it will be used for any new files created, for normal files this is normally 1, unless that file has been hard linked. for directors this changes depending on the number of subdirectories it contains, since each sub directory has a link back to the parent directory in the form of ‘..’, a directory without sub directories has 2, one for the parent directories link and another for the directories link to itself.

    We then have the username (root) which is associated with the owner permissions and then the group (also root) associated with the groups permissions. Then the time and date.

    Groups
    Users have a primary group but can belong to multiple supplemental groups. This is defined in the /etc/group file. This was its possible to have a file that one user can modify as their own, people in the same group as the file can read but not modify and everyone else is completely blocked. You can also use ‘usermod’ to change which groups a user belongs in. You can see what groups you are in with ‘id’

    For example:
    usermod -ag newgroup username
    The -a tells usermod to append the groups, without it any groups the user is in would be removed if they weren’t specified. -g is for secondary groups, normally these are all you only need to change.

    Mount points
    It is a good idea to have no permissions enabled for unmounted mount points such as /mnt/cdrom, you can then set another set of permissions for when it is mounted which will automatically be applied each time that file system is mounted. If you want regular uses to be able to mount something that is set in /etc/fstab not on the mount point permissions. Doing this will give users a permission denied error if they try to access an unmounted directory, rather than just getting an empty directory.

    letter mode vs octal letter permissions
    Often you will see chmod commands with number such as ‘chmod 750 /tmp/somefile’ these are permissions in octal mode (octal because there are 8 choices, 0-7), there is one number is for user,group and other. The numbers are a combination of the different permissions, each permission type is assigned a value, Execute is 1, write is 2 and read is 4, these numbers can them be added together to get a permission, such as 5 which is read and execute, or 7 which is full permissions. Sometimes there is a 4th number than is for the extra bits such as sticky and sudo. ‘man chmod’ for more information.

    If you don’t like the number system you can use the easier to remember letter system.
    Such as ‘chmod ugo+rwx’ which gives user, group and other full permissions.

    Setting permissions en mass
    You might want to set all files in a directory to one set of permissions such as 644, to allow user read and write, but everyone else read only. This can be done with ‘chmod -R 644 /directory’ but it has a problem, if you have sub directories and set these permissions the sub directories users will not be able to enter them because they need execute access. You can fix this with the command ‘chmod -R ugo+X /directory’, the capital X tells chmod to only apply executable bit on directories.





    Quick simple encrypted loopback filesystem

    16 04 2007

    UPDATE: For the extremely paranoid cryptoloop has a minor known vulnerability, it is possible to detect the presence of a specially created file in the file system, it shouldn’t cause a problem for most people. Wikipedia explains it here. If it is a concern for you then you might want something a bit more extreme. Cryptoloop has been superseded by dm-crypt, there is also truecrypt which can do 3 encryption algorithms at once and also hide your real encrypted fs under a dummy one. I’ll probably write up some more about them later.

    I’m doing the following under Debian Etch but Ubuntu Feisty should work exactly the same, other Linux distros might need the correct cryptoloop & aes encryption modules for the kernel and ‘/dev/loop0’ might change to ‘/dev/loop/0’, The kernel modules required for encryption are included in linux-image for both Debian and Ubuntu. As of Etch Debian also has fairly good support for encryption to be setup during the installation but thats not the goal of this article.

    Firstly you need to create a loop back image for your file system to use, the easiest way to do this is to output random data into a file, this will take a while for larger sizes, alternativly you could use /dev/zero but this could theoretically be less secure.
    dd if=/dev/urandom of=encrypted.img bs=1M count=100

    That will make a 100Mb loop back image, change the numbers to suit your desired size.

    Next we want to bind our image to a loopback device:
    losetup -e aes /dev/loop0 encrypted.img
    At this point you will be asked for a password, type it in. I recommend you use a longer pass phrase rather than a password, like an entire sentence with mixed case and numbers/symbols. When I was following some older how-tos i was getting an “ioctl: LOOP_SET_STATUS: Invalid argument”, that was because they had “-e AES256” rather than “-e aes”, with newer versions if you want to manually specify a keylength you use the -k flag although the default should be fine.

    Another common error is “ioctl: LOOP_SET_STATUS: Invalid argument” this is generally the result of not loading the correct modules, “modprobe aes loop cryptoloop” (actually cryptoloop will probally load everything required).

    Now we want to make a file system on it, works just like making one on a hard drive except we use the loopback device, most other howto’s I’ve seen on the subject seem to use ext2, on ext3 it is somewhat harder to recover deleted files (which could be a good thing for encrypted data) and you get journals that can help reduce/detect data loss. [EDIT: Now there is ext4 with things like extents which reduce fragmentation (although if your on a loopback device the loopback image will probably itself have some fragmentation too), journal checksumming, much faster fsck and other goodies (also btrfs but as of now it’s experimental).] Change ext4 to ext3, ext2 or btrfs if you want:
    mkfs.ext4 /dev/loop0

    Now we unbind the image from the interface:
    losetup -d /dev/loop0

    Now for automated mouting we want to make a mount point and edit the fstab:
    mkdir /mnt/encrypted

    To make sure that noone can access the mount point when the filesystem isn’t mounted, we need to set the correct permissions:
    chown 700 /mnt/encrypted

    The /etc/fstab entry should look like the following:
    /directory/with/image/encrypted.img /mnt/encrypted ext3 defaults,noauto,loop=/dev/loop0,encryption=aes 0 0

    Now try and mount it:
    mount /mnt/encrypted

    This should ask you for your password, if you get an error “mount: wrong fs type, bad option, bad superblock on /dev/loop/0,” you probably mistyped your password, otherwise you broke something.

    Make sure the permissions on the folder for its mounted state are correct, the following will only allow the user with you username to browse the directory (and root):
    chown -R username:username /mnt/encrypted
    chmod 770 /mnt/encrypted

    Remember that an encrypted system isn’t %100 secure, files can be cached in the swap drive, it is possible to encrypt swap space but you will loose some performance (Now days it might be better to just ensure you have plenty of ram and disable swap). Some programs will also cache information from the encrypted folder, such as file managers that make thumbnails for text/images/videos. It is possible to encrypt an entire installation, but generally there ins’t much point since files aren’t going to be saved in places like /usr/ and you will loose a lot of performance. It is also possible to use a encryption key file, rather than a password, this allows you to keep it on a USB drive but its possible the USB drive could be stolen too, or sized by the goverment tracking your MP3 downloading :p I would also recommend backing up the usb key in case you loose the key but not a laptop.

    If you want to increase the size of the filesystem later, firstly you need to add extra blocks to the file, to add 200Mb to the file:
    dd if=/dev/urandom bs=1M count=200 >> encrypted.img

    You can then resize the partition while it is mounted (its probably possible and best to do it when it isn’t but /dev/loop0 with losetup wasn’t doing it for me and increasing the size of an ext3 partition seems safeish and wasn’t throwing any warnings about it needing to be done offline):
    mount /mnt/encrypted
    resize2fs /dev/loop0





    Run a UNIX command on each file in a directroy

    28 03 2007

    I keep having to look it up at random times, so I though I would post it here.

    First there is the basic for loop, this however has some drawbacks, namely not being able to handle files with spaces in the filename, it sees a space as a newline and treats each individual part of the filename as an individual file. For the file “Sample File.txt” the the updating timestamp output is printed fine for the one file “Updating Sample File.txt”, but the touch command will execute on multiple files “Sample” and “File.txt”.
    for file in *; do echo "Updating timestamp for $file"; touch "$file"; done

    Using find with xargs, this only seems to be able to handle one command at a time (unless you bother to write a script for it). It puts all the commands on one line, rather than executing the command multiple times. It also doesn’t support more than one command, so echoing the message for each individual file seems impossible and it won’t work on commands unless they accept multiple filenames.
    find . -print0 | xargs -0 touch

    The next one is to use find with the -exec argument and some odd parameters, this allows for multiple command and spaces :):
    find . -exec echo "Updating timestamp for" {} \&\& touch {} \;

    Finally if you want something thats a bit nicer for use in scripts you can use read to stop the splitting of filenames:
    find . | while read FILE; do
    echo "Updating timestamp for $FILE"
    touch "$FILE"
    done;





    QEMU for DOS abandonware under Linux

    17 03 2007

    EDIT: System Shock 1 is now working for me under kqemu, see end.

    I decided I wanted to play some System Shock 1, unfortunately the game being several years old it no longer runs on modern system. There are emulation solutions such as DosBox (slow) and VDMSound (Windows only, I use Linux) however these have various flaws, bugs and compatibility issues, and even more problems when trying to run System Shock in particular. Unfortunately System Shock 1 wouldn’t work with sound but other games hopefully run better, the sound did work in Windows 3.1 🙂

    So I decided that I would give QEMU a try, its a nice fast Emulator since it also supports virtualization it can run at almost native speeds. Works on both Windows and Linux. And since we are emulating the entire computer rather than trying to intemperate the OS specific instruction sets we can get close to compatibility. We do however need to install an OS.

    These instructions are Linux based, however Windows also runs qemu so a lot of it is feasible, the main problem would be using the loopback floppy images.

    First of all we need our OS, a copy of DOS, there are a few free choices around such as FreeDOS and OpenDOS. Unfortunately these are not 100% compatible depending on what software you are running you will have mixed results, apparently System Shock has problems running on FreeDOS due to some custom memory management by the developers and I would guess the other clones would have similar issues also.

    There is also MS-DOS 7.1 which was the DOS shipped with Windows 98 repackaged, apparently that has some compatibility issues also.

    I decided to use MS-DOS 6.22, many people have an old copy of the floppy disks lying around. It’s probably not legal to download although googling for msdos622.zip shows many people think otherwise 😉

    Once you have gotten your copy of Dos, you will either have real floppies or probably IMA disk images (they might arrive in self extracting archives exe’s, under Linux you can just use unzip to extract the files form the exe, wine would probally work too) , these images are supposedly in a DiskWrite format however they appear to be just basic binary dumps of the floppies the same you would get from running a ‘dd if=/dev/fda of=floppy.img’. ‘file DOSDISK1.IMA’ shows them as “DOSDISK1.IMA: DOS floppy 1440k, x86 hard disk boot sector”. IMZ files are just zipped IMA files so just unzip them.

    If you are using original floppies, I recommend converting them to images with dd as images are a lot faster and floppies are fairly unreliable, its also handy to have backups.

    The next step will be to crate a image for the File System:
    qemu-img create -f qcow msdos.img 20G

    This will create a 20G image, because we are using the qcow format the image file will only take the amount of space as the data we install onto it. 20G is excessive, a FAT16 partition can’t use that much space but its not using any disk space so the size is irreverent and you don’t want to give yourself to little space as you would need to resize the partition or probably format and start again, the other format choice raw is more compatible but for the FAT file system it will take the entire amount of space you give it. (Trying the new qcow2 format might also be an idea, my version of qemu didn’t have it as an option though)

    If you are using disk images you will need to be able to access them as devices, run the following as root to bind them to /dev/loop0
    losetup /dev/loop0 DOSDISK1.IMA

    Now we need to boot the system with the qemu command (the faster ‘kvm’, Linux Kernal Virtualization Module version didn’t boot dos for me and crashed when Dos tried to initialize HIMEM (or directly after), I don’t have kqemu running so it might be worth disabling that if you have it installed and encounter problems. Stock qemu is slower but this stuff was designed to run on 33MHz so I don’t think there will be to much speed loss from the emulation layer, you can also use your real floppy drive with /dev/fda instead of /dev/loop0.
    qemu -soundhw sb16 -cdrom /dev/cdrom -fda /dev/loop0 -hda ./msdos.img -boot a

    You should get the MS-DOS 6.22 Setup screen, following the prompts it should partition and format the hard drive, reboot and then install.

    When the setup asks for Disk2, if you are using images press ctrl+alt+2 and you will be in the QEMU monitor mode. Type eject fda This disconnects Qemu from the loopback floppy image so you can disconnect the image from /dev/loop0.
    Pull up a terminal and losetup -d /dev/loop0

    Now you need to bind disk 2 to loop0:
    losetup /dev/loop0 DOSDISK2.IMA

    Go back to QEMU and type change fda /dev/loop0

    Press ctrl+alt+1 to resume the simulation and continue the Dos install procedure. Repeat for Disc 3.

    When DOS has finished installing it will attempt to boot from A again, just remove the ‘-boot a’ line to make it boot from the hard drive and you should end up in a dos prompt.

    Next is installing stuff. In order to transfer stuff from your local file system to the dos image, you need to run with the flag ‘-hdb fat:/your/driectory’, when I tried this I needed to be root. This will add an extra drive D: that maps to the directory specified, this allows you to put drivers and games in a directory and copy/install them onto the image under DOS. Otherwise you could try converting the image to a raw format (it will take the entire size of the image then), mounting on a loopback and copying the files across under Linux.

    You will want to track down the sb16 drivers to get sound working, google for “sbbasic.exe” and install the drivers under dos. And for mouse drivers CTMOUSE works fine (its from FreeDos) otherwise any other PS/2 dos mouse driver will do. You might want to add the mouse driver to your autoexec.bat so you don’t have to run it every boot

    There are also some video card drivers that allow Windows 3.1 to run in higher resolutions. Google “cirrus5446.zip”

    Now you can install and run games/programs.

    Unfortunately System Shock doesn’t detect the audio correctly (It works fine in Win 3.1). But hopefully other games will work better. EDIT: System Shock 1 is now working for me under kqemu with sound. Autodetect does fail sometimes but if you try several times it works. I’ve noticed that SoundBlaster Pro doesn’t crackle like SoundBlaster 16 does. Use Adlib for the music. Sometimes the sound isn’t detected by the game either so you might need to restart if there if you hear nothing on the main menu. When trying to install the SoundBlaster 16 drivers I was getting a “Program too big to fit in memory” error, I think this was caused by having ‘–swhardware all’ rather than just ‘–swhardware sb16’. I also had problems with it freezing and giving me a blackscreen leaving the mouse locked to the window (ctrl+alt not doing anything) and even after killing qemu from another console the mouse was not working requiring xorg to be restarted with ctrl+alt+backspace. I don’t know what was causing that but making a new hdd image and installing from scratch fixed it, maybe not having a copy of Windows 3.1 installed helped.

    Windows for Workgroups 3.11 under QEMU