Update on FreeBSD HAST Project

Pawel Jakub DawidekBack in October we mentioned that the FreeBSD Foundation decided the fund the HAST project (new FreeBSD Project: HAST)

Pawel Jakub Dawidek, the sponsored FreeBSD developer, has made quite a bit of progress since and finished the development of stage one:

I want to report that first milestone of the HAST project is complete.

Summary of the work that have been done:

  • Implementation of hastd daemon.
  • Implementation of hastctl utility to manage hastd daemons.
  • GEOM_GATE class was extended so that the caller can specify the name of GEOM provider. Before only /dev/ggateX names were supported. HAST will use /dev/hast/
  • Implementation of communication protocol. There is abstraction layer on top and below there are three protocols implemented currently:
  • proto_tcp4 – It is used for communication between primary and secondary nodes.
  • proto_uds – (UDS – UNIX Domain Socket) It is used for communication between hastctl and hastd.
  • proto_socketpair – It is used for communication between main hastd daemon and worker processes forked from it.
  • Implementation of nv (name-value) API, which allows to easy create packets containing name-value pairs. It is used for entire communication through the protocols above. It is also responsible for managing correct byte-order.
  • Implementation of ebuf (extendable buffer) API, which provides a way to extend given buffer by adding data at the back, but also at the front without reallocating it and copying the data very often (or never).
  • Implementation of logging API (pjdlog). The API decides if messages should be logged on standard output/error (before going into background) or to syslog (when we daemonize). It also provides some shortcuts for logging a message and exiting, etc. It supports notion of debug level and can skip messages intended for higher debug level than requested.
  • Implementation of configuration file parser in lex/yacc. Configuration file is designed in a way that it can be kept identical on both nodes.
  • Checksumming and compression for the data is not one of the project’s goal, but the stubs are there, so this can be added easly.
  • A lot of care was taken to be able to handle more nodes in the future. This is not implemented and in not project goal, but I wanted to make it ready for future improvements (source)

HAST enables users to use the FreeBSD operating system for highly available configurations where data has to be shared across the cluster nodes. The software will allow for synchronous block-level replication of any storage media (GEOM providers, using FreeBSD nomenclature) over the TCP/IP network and for fast failure recovery. HAST will provide storage using GEOM infrastructure, which means it will be file system and application independent and could be combined with any existing GEOM class. In case of a master node failure, the cluster will be able to switch to the slave node, check and mount UFS file system or import ZFS pool and continue to work without missing a single bit of data.

In order to fund more of these sort of projects, the Foundation is reliable on our donations and support. The Foundation hasn’t reached its 2009 goal yet. You can help by donating.

(BTW, I’m in no way affiliated with the work of the FreeBSD Foundation;  I only want to see FreeBSD progress to become even better)

FreeBSD end-of-year fund raising drive (update)

freebsd_foundation 100.x100FreeBSD is free; it can be downloaded, used and adapted without paying any fees, unlike other major operating systems. This is why the FreeBSD Foundation needs donations to be able to fund new projects and conferences.

The FreeBSD Foundation is collecting donations for the next year. The Foundation is a regular sponsor of many events, funds trips (including mine) to FreeBSD conferences and occasionally funds projects for various enhancements to FreeBSD. What else does the Foundation do?

The Foundation writes on its blog:

We would like to thank everyone who has donated to the FreeBSD Foundation this year. We have raised $183,888 towards our 2009 goal of $300,000! We are almost 2/3 of the way to reaching our goal! Oh, and BTW, we have had 671 donors this year. This is compared to just over 300 this time last year. This is important not only to help us keep our Public Charity Status, but it shows there are many users who are passionate about FreeBSD and want to show their support.

With the weakened economy we have been very conservative with our spending this year. But, like each previous year we have increased the amount we have spent on the FreeBSD Project and community. We were blown away with the number of project proposals we received this year. We were able to fund 7 projects this year. Unfortunately we didn’t have the budget to fund all the proposals we received.

This coming year we want to double the amount we spend on project development. In order to accomplish this, we need to meet our fund-raising goal.


Colin Percival has decided to donate all income made from his online backup service, Tarsnap, to the Foundation:

As a FreeBSD user and developer, I obviously care about the success of FreeBSD. I make a small contribution towards this success via my role as Security Officer; but the time I spend working on my Tarsnap online backup service prevents me from making as much of a direct contribution as I would like. Fortunately the FreeBSD Foundation does an excellent job of supporting FreeBSD development; but like most such organizations, they are funded entirely by donations and are always in need of more. In light of this, I am pleased to announce that I will be donating all of the profits made by Tarsnap for the month of December to the FreeBSD Foundation.


In memory of donations
After the passing away of John Birrell, the Foundation has made it possible to donate-in-memory-of:
It is now possible to make a donation to the FreeBSD Foundation in memory of someone. Select the “in memory of” button on the donation page and type in the name of the person the donation is being made in memory of.
Apart from my own donation, I donate 10% of the income (adverts, donations etc) from this blog.
Why not be generous, donate and help the Foundation reach it’s goal of $300.000 ?

Support FreeBSD and the FreeBSD Foundation

FreeBSD foundation logoMany people have already donated since my last post about the FreeBSD Foundation‘s end of year fund raising drive.

So far almost 600 people  (and companies) have donated to the FreeBSD Foundation, but the  $300.000 target hasn’t been achieved yet: the counter stands at $166.381.

What does the FreeBSD Foundation do?

We fund and manage projects, sponsor FreeBSD events, Developer Summits and provide travel grants to FreeBSD developers. We provide and help maintain computers and equipment that support FreeBSD development and improvements. We also represent the Project in executing contracts, license agreements, copyrights, trademarks, and other legal arrangements that require a recognized legal entity.

FreeBSD 8.0 has been released now. If you’re a happy FreeBSD user and want to support the development of FreeBSD 9, why not make a (small) donation? More people have donated this year than before. Any amount counts.

Donate quickly and securely


New FreeBSD Foundation Project: Flattened Device Tree

FreeBSD foundation logoThe FreeBSD Foundation has announced another funded project!

“Rafal Jaworowski and Semihalf has been awarded a grant to provide FreeBSD with support for the flattened device tree (FDT) technology. This project allows for describing hardware resources of a computer system and their dependencies in a platform-neutral and portable way.

The main consumers of this functionality are embedded systems whose hardware resources assignment cannot be probed or self-discovered.

The FDT idea is inherited from Open Firmware IEEE 1275 device-tree notion (part of the regular Open Firmware implementation), and among other deployments is used as a basis for Power.org’s embedded platform
reference specification (ePAPR).

Rafal JaworowskiThanks to this project, embedded FreeBSD platforms will grow in a uniform and extensible way of representing hardware devices, compliant with industry standards (ePAPR, Open Firmware), independent of architecture and platform (portable across ARM, MIPS, PowerPC etc.),

said Rafal Jaworoski, FreeBSD Developer.

Semihalf is a privately owned company, based in Krakow, Poland. They specialize in embedded systems design and development, with expertise in both software and hardware. Among their portfolio are FreeBSD ports to high-end embedded processors (including multi-core) with a wide range of peripheral drivers (storage, networking, pattern matching, security engines etc.); most of this work is publicly available from the FreeBSD repository.

You can find out more about the project at http://wiki.freebsd.org/FlattenedDeviceTree.

This project will complete by February 2010.”

If you want, you can support this project too.

New FreeBSD Foundation Project: HAST

FreeBSD foundation logoThe FreeBSD Foundation has announced that is funding a new funded project: HAST

“Pawel Jakub Dawidek has been awarded a grant to implement storage replication software that will enable users to use the FreeBSD operating system for highly available configurations where data has to be shared across the cluster nodes. The project is partly being funded by OMCnet Internet Service and TransIP BV.

The software will allow for synchronous block-level replication of any storage media (GEOM providers, using FreeBSD nomenclature) over the TCP/IP network and for fast failure recovery. HAST will provide storage
using GEOM infrastructure, which means it will be file system and application independent and could be combined with any existing GEOM class. In case of a master node failure, the cluster will be able to
switch to the slave node, check and mount UFS file system or import ZFS pool and continue to work without missing a single bit of data.

High-availability is the number one requirement for any serious use of any operating system,

Pawel Jakub Dawideksaid Pawel Jakub Dawidek, FreeBSD Developer.

Highly available storage is one of the key components in such environments. I strongly believe there are many FreeBSD users that have been waiting a long time for this functionality. I’ll do my best to deliver software that matches FreeBSD quality and that will satisfy the needs of our users.

Pawel has been an active FreeBSD committer since 2003. During this period, he has touched almost every part of the kernel. But, his main interest in FreeBSD is storage and security related topics. Pawel is the author of various GEOM classes (eli, mirror, gate, label, journal, hsec, etc.), geom(8) utility, various opencrypto improvements as well as port of the ZFS file system from OpenSolaris to FreeBSD.

The project will complete by February 2010.”

If you want, you can support this project too.

FreeBSD Foundation call for donations

FreeBSD foundation logoWe’re over half way through 2009, but the FreeBSD Foundation has not reached half of their 2009 fundraising goal. Justin Gibbs, founder and president of the FreeBSD Foundation, is calling on people’s generosity to support.

The Foundation is (part)funding some of the (Free)BSD conferences, sends developers to attend and funds new projects.

Justin writes:

Millions of systems run FreeBSD.  Hundreds of volunteers contribute to FreeBSD’s success.  But what is the size of FreeBSD’s user base?  This simple question is very hard to answer, but its answer is vital to the cause of promoting FreeBSD.  It is extremely difficult to convince
businesses to invest time and money to add FreeBSD support to their products based solely on vague estimates of the size of our community.
We should know – working to make FreeBSD a more widely supported platform is a task the FreeBSD Foundation has worked on since its inception.

Please help us in our fight to promote FreeBSD.  A donation to the FreeBSD Foundation helps fund our work, but it also gives us strength in numbers.  Our count of unique donors is a vital indication of the size and buying power of our community.  However, we have never broken even one thousand donors in any year.  We know in our hearts that this is a small fraction of our user base and of those who want to help expand FreeBSD’s presence.

So stand up and be counted!  Make a donation.  Encourage other FreeBSD users to donate as well.  No donation amount is too large or too small.  Just by becoming a donor you are making a powerful statement about the strength of FreeBSD!

You can make a donation by going to: http://www.freebsdfoundation.org/donate/.

To find out more about The FreeBSD Foundation, please visit http://www.freebsdfoundation.org.

FreeBSD foundation newsletter – June 2009

The FreeBSD Foundation have released their quarterly update. It gives a nice overview of the projects and conferences that are funded by the Foundation.
Table of contents:
  • Letter From the President
  • 2009 Fundraising Drive
  • Dru Lavigne Helping Foundation
  • Safe Removal of Active Disk Devices
  • Wireless Mesh Support
  • Improvements to the FreeBSD TCP Stack
  • AVR32 Support
  • Problem Reporting Prototype
  • FreeBSD Powers Long Distance Wireless Link
  • DCBSDCon 2009
  • AsiaBSDCon 2009
  • Foundation at BSDCan and Developer Recognition
  • 2009 Grant and Travel Grant Recipients
  • BSDCan Spotlight
  • Financials

Read the whole issue here.

The foundation is still way away from their donation target. To support FreeBSD and the FreeBSD Foundation, why not make a donation on their website? If you ever decide to support my website, 10% of your donation will be given to the Foundation.

Since recent times, the Foundation have their own blog and twitter account.

FreeBSD Foundation Project: FreeBSD terminal layer

The FreeBSD Foundation has announced another project that they’re funding:

Ed Schouten has been awarded a grant to write a new console driver for the FreeBSD project. We are excited to support Ed in providing a more efficient and user friendly console driver.

This project will allow Ed to add an additional abstraction layer to the kernel. This new layer, the terminal layer will be a layer that sits between the TTY layer, the kernel console (cngetc, cnputc) and the actual console driver. Right now we have a terminal emulator (libteken) that is part of Syscons. This terminal emulator will be moved into this
terminal layer.

The advantage of having such a layer, is that the console driver itself does not have to care about any TTY semantics, streams of bytes, processing escape sequences, etc. It will just receive a set of character drawing, filling and copying actions. This should also make it easier to implement Unicode.

“During this project I’m going to continue the work I did with the TTY layer, by developing a new console driver for the FreeBSD kernel,”

said Ed Schouten, FreeBSD Developer.

“By moving towards a graphics mode console driver, it will be much easier to make the boot process look nice on desktop systems (i.e. PC-BSD). It will also make it possible to support the industry-standard Unicode character sets by default.”

This project will be completed by the end of December.

Announcement: Two new projects funded by the FreeBSD Foundation

The FreeBSD Foundation has announced they have accepted two project proposals!

AVR32 – 32-bit MIPS Architecture

Arnar Mar Sig has been awarded a grant to develop AVR32 support for FreeBSD. AVR32 is a 32-bit MIPS architecture targeted for low power high throughput embedded applications. The target platform is the NGW100 reference design board from Atmel.

“This work will advance the MIPS support in FreeBSD and our capabilities in building embedded applications,”

said Sam Leffler, The FreeBSD Foundation, Director.

“I’m excited to be able to work on bringing FreeBSD to another architecture and pushing it farther into the embedded market,”

said Arnar Mar Sig, FreeBSD developer.

The project will be completed by August 2009.

FreeBSD Problem Reporting System

Mark Linimon has been awarded a grant to prototype a new problem reporting system for the FreeBSD project.  This project will allow Mark to define the features, look-and-feel, and architecture of a future replacement of the project’s current GNATs based system.  Once the prototype is complete, it will be used to garner input from the FreeBSD community before a production system is implemented.

Mark holds two positions within FreeBSD: one on the Ports Management team (portmgr) and one on the Problem Report Database administration team (bugmeister). He has also written the Ports Monitoring System to correlate data from the package building cluster, the Problem Report Database, the source control repository check-ins, and other sources.

“One of the most frequently requested improvements from the FreeBSD developer community is an improved bug tracking system,”

said Mark Linimon. He also added,

“The design goals of this prototype are to incorporate such features as markedly improved workflow, better categorization, customizable email notifications, and redesigned web pages to make searching and browsing easier. Once the prototype is completed,” Mark added, “it will be circulated amongst the developer community for feedback. I am happy to have the Foundation’s support to work on this project.”

“Problem reporting software is a critical tool for getting feedback from the FreeBSD user community, recording information about defects and missing features in the system, and making our volunteer developers productive,”

said Justin Gibbs, Founder of the FreeBSD Foundation.

“Mark has used manpower and sheer will to overcome the deficiencies in the current problem reporting system, and to make it work for the project.  But our GNATs isn’t fully utilized because of missing features and a clumsy user interface. We’re very excited to help address these problems in a core piece of the FreeBSD project’s infrastructure.”

This project will be completed by the end of June.

I’m sure everybody will join me to congratulate Mark and Arnar on their successful applications. We’re looking forward to AVR32 support for FreeBSD and a new FreeBSD bug reporting system.