2009 in review

Posted: Wednesday December 30, 2009 at 8:13 p.m.

So much for updating this blog more regularly. I guess this blog simply doesn't rank high on my list of priorities. Work has been, for lack of a better word, intense! Fortunately it has been so in a good way. There has been no shortage of challenges, most of which were technical, challenging and generally interesting. And the really good news is the work continues.

From a personal projects perspective, I've had to pick and choose which to run with. Failure to do so would have left me with a hollow list of ideas that never amounted to anything beyond entry level academic exercises. I have played around virtualization some early in the year using OpenVZ. I like the results, but have found it was not the good bet for my future use, as my work has started to turn to VMWare and Solaris Zones in a big way. Shortly after my post in February, I was able to complete an upgrade of the Django based websites. I am still quite pleased with Django, though I've not been keeping up to date on that scene very well.

I've spent most of my free time focused on Scala Bazaars (sbaz). I've been improving the sbaz code base in earnest only since June or so. I am a bit ashamed it took me so long. This is in part due to my needing to better familiarize myself with the existing code. The bigger reason was lack of vision, and therefore unsure direction. I spent more time contemplating what to do instead of doing. Sbaz has lost ground as a tool used by the public due to a variety of reasons. First and foremost has been the waning development of both code base and public repositories. I am attacking the former head on, and the later soon after. A more basic question is what use cases we should target. Most prospective users today are developers who have other tools that are more deeply integrated with IDEs, build tools, version management, etc. Should sbaz adapt to compete in the development space? Sbaz was modeled after the Linux package management tools, and I'm not convinced such a design can satisfy both the flexibility and consistency needs of developers unless we can establish some large scale release process similar to Linux distributions. My next post, if it happens within the next month or so, will likely be ramblings around some of these thoughts, and how I would like to see a sub-community start to form around sbaz.

Ultimately, I decided on a path of incremental improvements to existing functionality. This way I could contribute as little or as much as my existing family responsibilities and increasing work commitments would allow. My primary goals this far have included:

  1. Update to scala 2.8 and eliminate all compile warnings, including Java deprecations
  2. Introduce no backward incompatible changes (beyond minimum Java5 per 2.8 upgrade)
  3. Improve dependency audits to prevent a broken managed directory from ever appearing
  4. If an audit fails, make it clear to the user why
  5. Keep the user informed (specifically when downloading)
  6. Expand the body of automated testing
  7. Improve Windows support

These are still works in progress, but there has been good progress made. I have also added a few features like asynchronous downloads and support for pack200, the latter I'm amazed I haven't seen used more. I suppose the scala requirement of java5 or later makes such support easier..? At the time of this writing, there is still more to be done; however, the code base isn't far from being ready for the next major distribution release of Scala. I'd be working on it now if I wasn't away from my primary development machine.

scala   (0)

So many projects...

Posted: Sunday February 22, 2009 at 1:46 a.m.

... and so little time. I really should make a point of updating this blog more frequently, as I've been doing a lot of new and interesting (at least to me) technical stuff in the *nix space. Up until the end of last year, my day job has primarily focused on application capability. What I mean is I spent most of my time working on application specific configuration and code changes to add features or fix bugs to better support the business's needs. Most of these applications are large, complex enterprise tools that fall into the "one size fits all because you can customize through configuration" category. Granted, the line between code and configuration is unclear at times. On occasion, I would get down to the application framework or operating system level. It was always a treat.

Late last year, a key person for the company website's site operations team moved on, and I was pulled in to fill a fraction of the void he left behind. I'm only working there at half capacity because I still need to support my previous areas, but it has been a refreshing deviation from the work I've been doing for the past 7 or so years. And when I get interested and excited about something, I begin to explore.

Virtualization

One area I've spent some time exploring is system virtualization software, and this is for several reasons. I only have one machine at home to do all my exploratory work. This poses a problem when playing around with load balancing configurations or testing network software. Virtual servers allow me to simulate many machines on a single box, so while it isn't perfect for emulating a production environment, it makes it possible with a non-existent budget. Trying out different flavors of applications can also have a negative effect (sometimes damaging) on a box. Installing, configuring, testing and then uninstalling can lead to garbage being left behind. If the install is invasive to the basic services of a box and you don't fully understand all aspects of these changes, it can interfere with the health of the system in the long term. Using a virtual environment for this kind of playing around means you can simply discard the environment if you decide not to go with that solution. One added benefit of installing critical services into a virtual environment is you can easily back-up and transport that environment into another machine if the host box goes belly-up. Maybe I should get another machine...

Website - the whole package

I've had a few websites outside of work that I've developed (with someone else's code as a starting pointing) and maintain, this site being one of them. I've never put a proper backup/restore process into place for full, rapid restores. In the event of a failure, I would be able to restore most, but it would take a long time and some stuff may fall through the cracks.

Django 1.0 came out a while ago. I've successfully upgraded this site, but have some more upgrade work to do yet. This will likely take about 3 days to complete, if I could work on it full time. Unless I can use some dedicated time over a long weekend, it will probably grow into 3 weeks.

This site has only held my blog this far. I think I want to expand it to serve as my project site as well (of which I currently have none), as I've got bandwidth and storage space to spare. Source code will be hosted via mercurial, possibly through hgweb. I haven't decided if I want to use Trac. This partly depends on its RAM consumption because I need to keep several apps running on this hosting service, and memory is my limiting factor. I've already done some promising prototyping of a move from mod_python to mod_wsgi, so I appear to be making myself some wiggle room.

My Scala + db4o + Swing personal project

I've blogged about this project previously. I've succeeded in putting together a lot of the domain specific code this far, but I'm now toiling with the application framework (API for managing the main window, opening/closing views, starting and stopping services). I haven't decided on the user experience yet: single tabbed window, multi window, etc. The options are plentiful, but my goal is to keep it light weight with the potential for using it in a smart phone. I've started going down the path of OSGi (another learning experience). That may or may not continue. This is one project I would like to host out of this web site.

sbaz - Scala's package manager

I am presently the named maintainer for sbaz. Sbaz will start to become a bigger part of this blog. Presently, I am still struggling with defining a path forward for sbaz. I'm considering hosting some sbaz universes out of this site (e.g. my personal project above, a managed developer release universe, etc.). My biggest obstacle with this is my limited memory resources, so deploying the existing Java servlet won't be possible. Perhaps using PHP, Python or flat files to host a universe should be my first enhancement.

General system administration notes

I need a reliable location to document many of my system administration learnings, and this website seems to be a logical location. To date, I've peppered README files throughout my machine's filesystem, but the point behind the README files is to repeat the steps I need to perform on a new (e.g. replacement) machine. I've also failed to document some good stuff because I hope to, at some point, have a central repository. So, I delay documenting until I forget to do it all together, a pattern I've recently recognized and am changing... leading to the before mentioned README files. Some of the things I intend to document include:

db4o | misc. | scala   (0)

A new personal project

Posted: Friday June 6, 2008 at 10:59 p.m.

After coming to the disappointing conclusion that I cannot, at this time, pursue building my NAS device from scratch, I've turned toward another long contemplated and frequently revisited project. For quite some time I've dabbled with various Java based functionality, such as db4o, Scala and RCP frameworks, and have been toying with some ideas to implement a useful application from scratch.

I know it is pretty sad that a professional software developer of 7 years is psyched about creating an application from scratch, but most of my development work involves configuring and/or extending existing applications purchased from third party vendors, identifying and fixing bugs in code others have written, addressing performance issues, and interfacing multiple enterprise applications. I have done very little UI development, and most of that has been for web applications.

Standing at the start of my desktop application exploration, I realize my first goal is to implement something simple and useful with an expandable design. The tool needs to be useful to keep my interest, otherwise the application will be yet another academic exercise. The simple but useful feature provides the bait that leads me along, but by having an extensible design, adding additional features becomes incremental growth instead of titanic rewrites and redesigns. So, I have the following thoughts:

  1. The application should run on the JRE and use the Scala programming language for at least part of the functionality. I am learning to love functional programming, but haven't applied it to any practical application yet.
  2. If using a database, I would like to use db40 instead of a relational database. A lot of the stuff that slows down development with a database goes away when the Java class IS the database schema. No dual implementation (Java class and relational schema) or mappings are required. Of course, I may be better off persisting my data as flat files.
  3. I should standardize on a single IDE (Eclipse or NetBeans), as this is primarily a single developer effort, and trying to develop in two different IDEs would require a lot of overhead. Unfortunately, this adds limitations to the technology I can use for development. For example, Scala only has a plug-in supporting Eclipse, I am more familiar with Eclipse's features, I like the idea of learning more about OSGI, and I like the responsiveness of SWT vs. Swing. On the flip side, only NetBeans has Matisse (for free anyway), the profiler seems nicer, RCP appears to be easier, I'm more familiar with the Swing API, I like the idea of a single distribution for all machines (SWT requires different DLLs), and I LOVE the mercurial plug-in.
  4. I don't expect this code will be opened for general availability, but I should take legal considerations (GPL, LGPL, EPL, BSD, etc.) into account when choosing libraries. Otherwise, the conflicting licenses could become prohibitively difficult to work around.
  5. I would like to leverage a multi-document interface (MDI), at least optionally, with tabs that can be reordered, maximized to full screen, closed with middle button click, and possibly minimized to border buttons.
  6. Ideally, data could be replicated between machines with ease, as I would like to use the application on my home and work machines. This, of course, requires multi-platform functionality, as my work laptop runs Windows XP while my home machine runs Ubuntu Linux. (Gosh how I wish I could run Ubuntu at work too...)

Clearly, I have more to think about before choosing an IDE. As for the applicaiton's features, I've considered the following:

  1. A secure password store
  2. A wiki style notebook similar to Tomboy Notes for Linux. I found that taking notes in ASCII with a markup "language" like markdown worked quite well. The file's name contains the timestamp of when the file was created, and all context of what the note was about is contained within. When the file is stored in the file system, the operating system's indexing tool can be leveraged for searches, and the application can manage categorization in a flexible way. I'd like to include IDE style tab completion for internal links, syntax highlighting, daily todo list feature, hyperlinks outside the application (e.g. web pages, lotus notes documents, network share locations, etc.) and more.
  3. A batch image resize tool for high quality images. I've actually already implemented this using the ImageJ library available in the public domain, but the GUI is a bit hackish and not robust.
  4. A contact management application that leverages low level relationships. For example, a family of five may all have the same contact information while the kids are young, but as each kid gets his/her own email address, mailing address (e.g. goes to college), cell phone, etc., each individual's contact information can be updated accordingly. Or, if the entire family moves to a new location, a single change to their mailing address node would update all family members at once. This would be a perfect application for db4o.
  5. An invoicing and general business tracking application for my wife to use for her photography business. This is a nontrivial piece of functionality with the potential to grow significantly. It could be used to drive marketing, schedule photo shoots, track income and expenses and so on.

I really need to have an IDE that makes the GUI layout and interaction with the business logic clear and easy to do in order to implement all that I would like. I suspect this seem daunting mostly because it is new to me. I'm still looking for some best practices on how to layout the MVC design of the GUI, though I suspect I'll come to a solid conclusion only after I've tried it.

db4o | Java | NAS | RCP | scala   (0)

Scala and db4o Native Queries

Posted: Wednesday November 28, 2007 at 11:51 a.m.

Recently, I've been playing with Scala, a multiple paradigm programming language that runs within the Java virtual machine. While searching for on-line resources describing practical applications of the language, how-tos and best practices, I ran across a blog entry by N. Chime showcasing the object database db40. I immediately felt a rush of excitement. I have been keeping an eye on db4o for years but have not had the opportunity to use it beyond a few short-lived pet projects or explorations of new-to-me technology (e.g. Wicket). The technology has always interested me, and I've been hoping to use it in a real application.

Using Chime's blog as a road map, I started creating a contact management app with Scala and db4o. Yes, this is yet another exploratory project, but I hope to evolve it into something real. Anyhow, everything was coming together fine until I tried to implement db4o's native queries. No matter what I did in the filter method of the Predicate class, I was unable to get the expected results. As it turns out, there is a bit of a Catch-22 when using this feature.

Here you can see an example of a native query taken directly from Chime's blog entry. You can trust that the Pilot class and listResult(...) function work as expected. I will provide fully functioning code later, but I want to show the obvious/intuitive implementation and how it fails. 

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(point : Any) : boolean = {
      val p = point.asInstanceOf[Pilot];
      return (p.getPoints > 99) &&
             (p.getPoints < 199) &&
             (p.getName.equals("Rubens Barrichello"))
    }
  });
  listResult(result);
}

Note that I have changed the match method's input parameter from All, as shown in Chime's entry, to Any, the Scala equivalent to java.lang.Object. Without doing this, the Scala compiler complains because the Predicate class defines the abstract match method with a generically typed parameter. Scala does not support Java generics, so more restrictive typing is lost in translation. That is, Java has generics and Scala has generics, but they don't work together yet. As a result, the Scala compiler will complain if you try to implement the match method with any type other than Any. I suppose more restrictive type checking of the compiler was introduced through the evolution of the Scala language, which is why the All type (includes null) no longer works.

My attempts at using this implementation failed. I started using the db4o-6.3-java5.jar, and found that my native queries failed at run time with a java.lang.IllegalArgumentException due to invalid predicate. With a little digging on-line, I found this error is generated when the Predicate does not have a properly defined filter method: the boolean returning method named "match" that accepts a single parameter. The above Predicate implementation does contain a "match" method that returns a boolean and accepts a single parameter, so I considered that Scala's missing support of Java generics could be throwing a wrench into things. I switched the db4o implementation to db4o-6.3-java1.2.jar, which is intended for Java 1.2 through 1.4, to remove generics from the equation. This eliminated the exception; however, the queries failed to return correct results. No matter what I did to the "match" method, even hard coding boolean return values, the database behaved as if the method always returned the same value. Implementing the same queries in Java yielded the expected query results, so this issue was clearly a problem with the Scala code.

And the hero of the day is... open source! I finally got around to looking at the db4o source code to find the root cause of my issue, and it didn't take long to find the problem within the Predicate.getFilterMethod() method. The code explicitly ignores any match method that accepts a single java.lang.Object parameter. While the db4o documentation clearly states that the match method must take one parameter, it fails to mention that this parameter cannot have the general java.lang.Object type.

So the Scala compiler requires that the match method accepts a parameter typed as Any (effectively an alias for java.lang.Object) while db4o explicitly ignores such a parameter. The workaround is to provide both. 

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(point : Any) : boolean = 
        throw new Exception("This should never be called!")
    def `match`(p : Pilot) : boolean = {
      return (p.getPoints > 99) &&
             (p.getPoints < 199) &&
             (p.getName.equals("Rubens Barrichello"))
    }
  });
  listResult(result);
}

The above implementation will generate the expected results because:

  1. The Scala compiler is satisfied by the stub method accepting the Any type
  2. The db4o reflection logic can find a match method with a more constrained parameter type

Certainly this code can better conform to the DRY principle by pulling the stub method into a custom abstract class. For your viewing and testing pleasure, here is the complete code for my working native queries test of the db4o object database with Scala. While I have improved DRY for the stub match method, I'm sure more could be done for converting the query returns into native Scala Iterators (Read: It's late and I want to get this post up before going to bed). 

package db4osc.chapter1;

import com.db4o._;
import com.db4o.query._;
import scala.collection.jcl.MutableIterator

object NQExample extends Application with Util {

val db = Db4o.openFile("chapter1.db");

  storePilots(db);
  retrieveComplexSODA(db);
  retrieveComplexNQ(db);
  retrieveArbitraryCodeNQ(db);
  clearDatabase(db);

  db.close();

  def storePilots(db : ObjectContainer) = {
    db.set(new Pilot("Michael Schumacher",100));
    db.set(new Pilot("Rubens Barrichello",99));
  }

  def retrieveComplexSODA(db : ObjectContainer) = {
    val query : Query = db.query();
    query.constrain(classOf[Pilot]);

    val pointQuery : Query = query.descend("points");
    query.descend("name").constrain("Rubens Barrichello")
      .or(pointQuery.constrain(99).greater()
        .and(pointQuery.constrain(199).smaller()));
    val result = query.execute();
    listResult(SIterator(result));
  }

  def retrieveComplexNQ(db : ObjectContainer) = {
    val result = db.query(new Filter() {
      def `match`(p : Pilot) : boolean = {
        return (p.points > 99) &&
               (p.points < 199) &&
               (p.name.equals("Rubens Barrichello"))
      }
    });
    listResult(SIterator(result));
  }

  def retrieveArbitraryCodeNQ(db : ObjectContainer) = {
    val points = Array(1,100);
    val result = db.query(new Filter() {
      def `match`(p : Pilot) : boolean = {
        return (p.points == 1) || (p.points == 100)
      }
    });
    listResult(SIterator(result));
  }

  def clearDatabase(db : ObjectContainer) = {
    val result : ObjectSet = db.get(classOf[Pilot]);
    while(result.hasNext()) {
      db.delete(result.next());
    }
  }
}

case class SIterator[A](underlying : ObjectSet) extends CountedIterator[A] {
  def hasNext = underlying.hasNext
  def next = underlying.next.asInstanceOf[A]
  def remove = underlying.remove
  def count = underlying.size
}


abstract class Filter extends Predicate() {
  def `match`(dummy:Any) : boolean = throw new Exception("Not supported") 
}


trait Util {
  def listResult(result : Iterator[Any]) : Unit = {
    println(result.counted.count);
    result.foreach(x => println(x))
  }
}


class Pilot(val name : String, var points : Int) {
  def points_+(apoints : Int) : Int = {
    points = points + apoints;
    return points;
  }
  override def toString() : String = return name+"/"+points;
}

Executing this should result in the following output. The integer indicates the first row of output for each result set, and corresponds to the number of hits from each query. 

2
Michael Schumacher/100
Rubens Barrichello/99
0
1
Michael Schumacher/100

I suspect the evolution of the Scala programming language led to the original code that Chime implemented becoming invalid code. Hopefully, when Scala gains support for Java generics, the need for this workaround will go away, and a more intuitive solution will work as expected. This was successfully implemented with:

EDIT November 29, 2007 -- I got to thinking that the above solution makes perfect sense for the Java 5 version of the db4o library, but makes no sense for the version that does not support generics. So, I revisited my implementation with db4o-6.3-java1.2.jar, and found that this will behave correctly when implemented in the obvious way. I was running into trouble because my match method was still implemented with the Any type. When I changed the code to use a more constrained type, it just worked. For example: 

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(p : Pilot) : boolean = {
      return (p.points > 99) &&
             (p.points < 199) &&
             (p.name.equals("Rubens Barrichello"))
    }
  });
  listResult(SIterator(result));
}

In this case, my trouble was due to the undocumented (at least in the API) requirement that the java.util.Object is an invalid parameter type for the match method. Still, I'm surprised I did not get an exception. If I get some time and motivation, maybe I'll dig into it more.

db4o | scala   (2)

by tag

recent posts

by date

feeds

recent comments