Project names are very important to me in a “the clothes make the man” sensibility.  I’d prefer to leave an application untitled rather than have a contrived or pedestrian name attached to it (which is why my EAD publisher app never had a title — nothing non-EAD-derogatory ever came to me).

Often, the name is the only part of a project that “works” (see:  FancyPants, CommuniCat etc.), so, in my mind, it’s important that it’s memorable enough that people (coworkers, mostly) pay attention to the initial pitch so you don’t have to explain its functionality every time.  When you are in the brainstorming/gathering-enough-interest-to-get-the-green-light phase, everything’s about marketing.  And, for me, that means a good working title.  I don’t like acronyms, usually, because I don’t think they’re terribly interesting (WAG the Dog’s acronym notwithstanding - I liked that one).

Anyway, “Jangle” was there before I was.  I don’t have to like the name to think the project is worthwhile.  Still, sometimes it pains me to spend all my time on a project that I had no influence over the title.  Silly, yes, but that’s why my pants are so fancy.

I first ran into my big XML and Ruby problem a couple of weeks ago while working on the TalisLMS connector for Jangle.  I’ve, of course, run into it before, but it has never been a total show stopper like this.  In order to add the Resource entity to the TalisLMS (Jangle-ese for bibliographic records) connector, I am querying the Platform store the OPAC uses.  I’m using the Platform rather than the Zebra index that comes with Alto (the records are indexed in both places) because the modified date isn’t sortable in Zebra and that would be an issue when serializing everything to Atom.  The records are transformed into a proprietary RDF format (called BibRDF) when loaded into the Platform (this is for the benefit of Prism, our OPAC).  In order to get the MARC records (there’s no route back to the MARC from BibRDF), I have to pull the UniqueIdentifer (which is the mapped 001)  field out of the BibRDF and throw them in a Z39.50 client (Ruby/ZOOM) and query the Zebra index.  In order to get enough metadata to create a valid Atom entry, I needed to be able to parse the BibRDF (which comes out of the Platform as RDF/XML), since that is the default record format.

And this is where I’d run into problems.  I have the default number of records set to be returned by the Jangle to 100.  That’s a pretty sweet spot for both servers to handle the load and clients to deal with resulting Atom document.  Well, you’d think it was, anyway, except REXML was taking about 10 seconds to parse the Platform response into Ruby objects.

I realize the Rubyists out there are already dismissing this and scrolling down to the comment box to write “well don’t use REXML, you dumbass”, but let me explain.  I generally don’t use REXML (unless it’s something very small and simple), instead opting for Hpricot for parsing XML.  I’ve tended to avoid LibXML in Ruby, when I first tried it, it segfaulted a lot, but that was the past… my reasons for avoiding it lately is because I have this stubborn ideal about having things work with JRuby and that’s just not going to be an option with LibXML (before you scroll down and add another comment about the Ruby/ZOOM requirement, it will eventually be replaced with Ruby-SRU… probably).  Hpricot was falling flat on its face with the BibRDF namespace prefixes, though (j.0:UniqueIdentifier).  It seems to have problems with periods in the prefix, so that was a no go.

So I had REXML and I had horrible performance.  Now what?

Well, JSON is fast in Ruby, so I thought that might be an option.  The Platform has a transform service, if you pass an argument with the URL for an XSLT stylesheet, it will output the result in the format you want.  Googling found several projects that would turn XML into JSON via XSLT (this one seems the best if you have an XSLT 2.0 parser), but they weren’t quite what I needed.  I wanted to preserve the original RDF/XML since I was just going to be turning around and regurgitating it back to the Jangle server, anyway.  I just needed a quick way to grab the UniqueIdentifier, MainAuthor and LastModified fields and shove the rest of the XML into an object attribute.

I have always chafed at the thought of actually doing anything in XSLT.  In retrospect (after I’ve been using almost exclusively for a month, now), I realize that my opinion was probably actually the result of the data that I was trying to transform (EAD, the metadata format designed to punish technologists) rather than XSLT itself (the project got sucked into a vortex when I tried working with the EAD directly with Ruby, too).  Still, I had always resisted.  The syntax is weird, variables confused me, I just never got the hang of it.

But, damn, it’s fast.

And, when I turned the XML into JSON (with XML), it was perfect.  Here’s my stylesheet.  Here’s what the output from the Platform looks like.  Here’s the output from the TalisLMS connector.

I wasn’t done, yet, though.  The DLF ILS-DI Adapter for Jangle’s OAI-PMH service was sooooo slow.  Requests were literally taking around 35 seconds each.  This was because I was using FeedTools to parse the Atom documents and Builder::XmlMarkup to generate the OAI-PMH output.  And this was silly.  Atom is a very short hop to OAI-PMH, and there was really no need to manipulate the data itself at all.  However, I did need to add stuff to the final XML output that I wouldn’t know until it was time to render.  So I wrote these two XSLTs.  I have patterns in there which are identified by “##verb##” or “##requestUrl##”, etc.  This way, I can load the XSLT file into my Ruby script, replace the patterns with their real values via regex, and then transform the Atom to OAI-PMH using libxslt-ruby.  Requests are now down to about 5 seconds.  Not bad.

All in all I’m pretty happy with this.  And I don’t have to quit my addiction just yet.

For those of you that noticed that libxslt-ruby doesn’t quite jibe with my JRuby requirement, well, I guess I’m not a very dogmatic at the end of the day (which is right about now).

The skepticism of Jangle is understandable and expected.  After all, it’s a very unorthodox approach to library data, seemingly eschewing other library initiatives and, at the surface, seems to be wholly funded by a single vendor’s support.

And, certainly, Jangle may fail.  Just like any other project.  Just like VuFind.  Just like Evergreen.  Any new innovative project brings risk.  More important than the direct reward of any of these initiatives succeeding is the disruption they bring to the status quo.  Instead of what they directly bring to the table, what do they change about how we view the world?

Let’s start with Evergreen.  Five years ago I sat in a conference room at Emory’s main library while Brad LaJeunesse and Jason Etheridge (this predated PINES hiring Mike Rylander and Bill Erickson) told us that they were ditching Unicorn and building their own system.  I, like the others in the room, Selden Deemer, Martin Halbert, smiled and nodded and when they left I (Mr. Library Technology Polyanna) turned to the others and said that I liked their moxie, but it was never going to work.  Koha was the only precedent at the time, and, frankly, it seemed like a toy.

Now where are we?  Most of the public libraries in Georgia using Evergreen, a large contingency from British Columbia migrating, and a handful of academic libraries either live or working towards migration.  Well, I sure was wrong.

The more significant repercussion of PINES going live with Evergreen was that it cast into doubt our assumptions of how our relationship with our integrated library system needed to work.  Rather than the library waiting for their vendor to provide whatever functionality they need or want, they can instead, implement it themselves.  While it’s unrealistic for every library to migrate to Evergreen or Koha, these projects have brought to light the lack of transparency and cooperation in the ILS marketplace.

Similarly, projects like VuFind, Blacklight and fac-back-opac prove that by pulling some off-the-shelf non-library-specific applications and cleverly using existing web services (like covers from Amazon) that we can cheaply and quickly create the kinds of interfaces we have been begging from our vendors for years.  It is unlikely that all of these initiatives will succeed, and the casualties will more likely be the result of the technology stack they are built upon rather than any lack of functionality, the fact that they all appeared around the same time and answer roughly the same question, shows that we can pool our resources and build some pretty neat things.

To be fair, the real risk taker in this arena was NC State.  They spent the money on Endeca and rolled out the interface that wound up changing the way we looked at the OPAC.  The reward of NCSU’s entrepreneurialism is that we now have projects like VuFind and its ilk.  Very few libraries can afford to be directly rewarded by NC State’s catalog implementation, but with every library that signs on with Encore or Primo, III and Ex Libris owe that sale to a handful of people in Raleigh.  You would not be able to download and play with VuFind if NC State libraries had worried too much about failure.

Which then brings me to Jangle.  The decision to build the spec on the Atom Publishing Protocol has definitely been the single most criticism of the project (once we removed the confusing, outdated wiki pages about Jangle being an Rails application), but there has been little dialogue as to why it wouldn’t work (actually, none).  The purpose of Jangle is to provide an API for roughly 95% of your local development needs with regards to your library services.  There will be edge cases, for sure, and Jangle might not cover them.  At this point, it’s hard to tell.  What is easier to tell, however, is that dwelling on the edge cases does absolutely nothing to address the majority of needs.  Also, the edge cases are mainly library-internal-specific problems (like circulation rules).  A campus or municipal IT person doesn’t particularly care about these specifics when trying to integrate the library into courseware or some e-government portal.  They just want a simple way to get the data.

This doesn’t mean that Jangle is solely relegated to simple tasks, however.  It just is capable of scaling down to simple use cases.  And that’s where I hope Jangle causes disruption whether or not it is ultimately the technology that succeeds.  By leveraging popular non-library-specific web standards it will make the job of the systems librarian or the external developer easier, whether it’s via AtomPub or some other commonly deployed protocol.

Maybe it’s about time you all started to explore this idea again.

So far, it’s been pretty ugly.  I haven’t had much time to work on it, granted, but the time I’ve spent on it has made me think that there will be a lot of work to do.  Couple this with some of the things that make the Platform difficult to work with in Ruby anyway (read: Digest Authentication) and this might be a more uphill battle than I’ll ever have time for, but I figure it’s either this or go back to Python and I’m not quite ready to give up on Ruby yet.

Currently, performance is abysmal with ActiveRDF against the Platform, so I’ll need to think of shortcuts to improve that (I’m not even considering write access presently).  Here’s some code (this is as much for my benefit, so I can remember what I’ve done) to work with Ian Davis’ Quotations Book Example store:

require ‘time’ # Otherwise ActiveRDF starts freaking out about DateTime
require ‘active_rdf’

$activerdf_without_xsdtype = true
# less than ideal, but without it, ActiveRDF sends
# ^^<http://www.w3.org/2001/XMLSchema#string> with string literals even if you don’t want
# to send the datatype.  I haven’t actually tried it with other datatypes to see how this breaks
# down the road.

ConnectionPool.set_data_source(:type => :sparql, :results => :sparql_xml, :engine=>:joseki,  :url=> “http://api.talis.com/stores/iand-dev2/services/sparql”)

Namespace.register :foaf, “http://xmlns.com/foaf/0.1/”
Namespace.register :dc, “http://purl.org/dc/elements/1.1/”
Namespace.register :quote, “http://purl.org/vocab/quotation/schema”

QUOTE::Quotations.find_by_dc::creator(”Loren, Sophia”).each do | quote |

# print the important stuff from each graph

# http://purl.org/vocab/quotation/schema#quote has to be manually added as a predicate
# the “#” seems to cause problems
quote.add_predicate(:quote, QUOTE::quote)
puts quote.quote
puts quote.subject
puts quote.rights
puts quote.isPrimaryTopicOf

end

If you actually try to execute this, you’ll see that it takes a long time to run (God help you if you try it on QUOTE::Quotations.find_by_dc::subject(”Age and Aging”)).  A really long time.

If you set some environment vars before you go into irb:

$ export ACTIVE_RDF_LOG_LEVEL=0
$ export ACTIVE_RDF_LOG=./activerdf.log

then you can tail -f activerdf.log and see what exactly is happening.

After ActiveRDF does it’s initial SPARQL query (SELECT DISTINCT ?s WHERE { ?s <http://purl.org/dc/elements/1.1/creator> “Loren, Sophia” . }), it’s doing two things for every request in the block:

1. a SPARQL query for every predicate associated with the URI (http://api.talis.com/stores/iand-dev2/services/sparql?query=SELECT+DISTINCT+%3Fp+WHERE+%7B+%3Chttp%3A%2F%2Fapi.talis.com%2Fstores%2Fiand-dev2%2Fitems%2F1187139384317%3E+%3Fp+%3Fo+.+%7D+)
2. a SPARQL query for the value of the attribute (predicate):  http://api.talis.com/stores/iand-dev2/services/sparql?query=SELECT+DISTINCT+%3Fo+WHERE+%7B+%3Chttp%3A%2F%2Fapi.talis.com%2Fstores%2Fiand-dev2%2Fitems%2F1187139384317%3E+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2Fcreator%3E+%3Fo+.+%7D

for every predicate in the graph.  You can imagine how crazily inefficient this is, since to get every value for a resource, you have to make a different HTTP request for each one.

Obviously this would be a lot easier if it used DESCRIBE rather than SELECT, but without a real RDF library to parse the resulting graph, I’m not sure how ActiveRDF would deal with what the triple store returned.

So, anyway, these are some of the hurdles in making ActiveRDF work with the Platform, but I’m not quite ready to throw in the towel, yet.

Sorry about that.

When I was still at Georgia Tech, one of the things I was trying to work on was a framework to consistently and easily expose the library’s data from its various silos into external services. In that case, my initial focus was the Sakai implementation that we were rolling into production, but the intention was to make it as generic as possible (i.e. the opposite of a “Blackboard Building Block“) so it could be consumed and reconstituted into as many applications as we wanted.

Coincidentally (and, for me, conveniently), Talis was also thinking about such a framework that would supply a generic SOA layer to libraries (and potentially beyond) and contacted me about possibly collaborating with them on it as an open source project. Obviously that relationship changed a bit when they hired me and they put me and my colleague Elliot Smith (reports of his demise have been greatly exaggerated) in charge of trying to get this project off the ground. Thankfully, Elliot is the other Talis malcontent who prefers Ruby, so our early prototypes all focused on Rails (the Java that originally seeded the project, like all Java, made my eyes glaze over).

We had a hard time getting anywhere at first. Not even taking into consideration the fact that he and I were an ocean apart, we really had no idea what it was that we should be building or why it would be useful to Talis (after all, they are paying the bills) since they already have an SOA product, Keystone. Also, we didn’t want to recreate Apache Synapse or Kuali Rice. In essence, we were trying to find a solution to a problem we hadn’t really defined, yet.

In December and early January, I drove across town for a couple of meetings with Mike Rylander, Bill Erickson and Jason Etheridge from Equinox to try to generate interest in Jangle and, at the same time, solicit ideas from them as to what this project should look like and do. Thankfully, they gave me both.

Jangle still foundered a bit through February. We were waiting for the DLF’s ILS and Discovery Systems API recommendation to come out (since we had targeted that as goal) and Elliot produced a prototype in JRuby (we had long abandoned Rails for this) that effectively consumed the Java classes used for Keystone and rewrote them for Jangle.  The problem we were still facing, though, is that we were, effectively, just creating another niche library interface from scratch and there were too many possible avenues to take to accomplish that.  Our freedom was paralyzing us.

I gave a lightning talk on Jangle at Code4lib2008 that was big on rah-rah rhetoric (free your data!) and short on details (since we hadn’t really come up with any yet) that generated some interest and a few more subscriptions to our Google Group.  A week later, the DLF met with the vendors to talk about their recommendation.   I attended by phone.  While in many ways I feel the meeting was a wash, it did help define for me what Jangle needed to do.

At the end of my first meeting with Equinox, Mike Rylander asked me if we had considered supporting the Atom Publishing Protocol in Jangle.  At the time, I hadn’t.  In fact, I didn’t until I sat on the phone for 8 hours listening to the vendors hem and haw over the DLF’s recommendation.  The more I sat there (with my ear getting sore), the more I realized that AtomPub might be a good constraint to get things moving (as well as useful to appealing to non-library developers).

We are just now trying to start building how this spec might work.  Basically there are two parts.  First, the Jangle “core” which is an AtomPub interface to external clients.  It’s at this level that we need to model how library resources map to Atom (and other common web data structures, like vCard) and where we need to extend Atom to include data like MARC (when necessary).  The Jangle core also proxies these requests to the service “connectors” and translates their responses back to the AtomPub client.  The connectors are service specific applications that takes the specific schema and values in, say, a particular ILS’s RDBMS and puts them in a more syntax to send back to the Jangle core.  Right now, the proposal is that all communication between the core and connectors would be JSON over HTTP (again, to help forward momentum).

So at this point you may be asking why AtomPub rather than implementing the recommendations of the DLF directly?  The recommendation assumes the vendors will be complicit, uniform and timely in implementing their API and I cynically feel that is unrealistic.  I also think it helps to get a common, consistent interface to help build interoperability (like the kind that the DLF group is advocating), since then you’d only have to write one, say, NCIP adapter and it would work for all services that have a Jangle connector.  Also, by leveraging non-library technologies, it opens up our data to groups outside our walls.

So, if you’re interested in freeing your data (rah-rah!), come help us build this spec.  We’re trying to conform to the Rogue ‘05 specification that Dan Chudnov came up with for development of this so, while it will still be a painful process, it won’t be painful and long. :)  In other words, this ain’t NISO.

Why would I subject myself to this? Well, a majority of my family lives in Portland and it seemed like a good opportunity for Che to meet his 97 year old great-grandmother. Since Selena couldn’t go, my mother volunteered to join me and take care of Che while I was in the conference. Surprisingly, despite ridiculously planned twelve hour traveling days, Che was a remarkably good traveling companion. Especially given that the day before we left, he was up all night vomiting from a stomach virus he picked up at daycare. But enough about my stupid traveling decisions for now.

Code4lib 2008 set an awfully high bar for Providence, RI to follow. Jeremy Frumkin managed to find an amazing hotel with both free breakfast and happy hour in the downtown of a remarkable city and somehow managed perfect weather in Portland at the end of February. Is it ok to name Jeremy president for life of Code4lib? Good luck, Brown University, Jeremy just made your job insanely hard.

I would say the overarching theme of this year’s event was “help us with our open source project!”. Or maybe, “fooled you! This actually all about RDF”.   Also, given how this is one of the more premier library technology conferences, we might want to give a scholarship to an MBA or two to teach us how to use a projector and Powerpoint.  I began to think there was a hex on the podium.

Rather than go over the presentations themselves (since that will be done more eloquently elsewhere, I’m sure), I’ll just make some observations. My perspective changed somewhat after Brewster Kahle’s keynote and Rob Styles’ presentation (during the following break) since my role in the conference went from ‘jerk in the audience making snarky comments in IRC’ to master of ceremonies when Jeremy had to go back to Corvallis to care for his sick daughter (or, rather, for his sick daughter and exhausted wife).

Rob was the first to focus on RDF and I think the constraint of our twenty minute presentation slots was both a blessing and a curse here. Rob did an excellent job of explaining why RDF is a good fit for MARC data (or, rather, the metadata that we are currently putting in MARC), but there wasn’t enough time for that and demonstrating why we would want to go through the effort of actually doing it (like, for linked open data and whatnot). It was a good overview of busting the MARC into its conceptual components and making connections between those components, however.

If any of my former colleagues at Georgia Tech still read my blog (hello!), I definitely urge you to take a look at Oregon State’s Interactive Course Assignment Pages (ICAP). This is exactly what I was trying to do when I wrote the subject guides application there. There is a lot of what appears to be NIH syndrome when it comes to subject/course guide applications; in reality I think these applications have to conform to a lot of local needs and expectations which is why “somebody else’s code” doesn’t always do the trick. ICAP is so similar to what Tech is running (although executed much, much more elegantly) that I don’t think there would be much of a leap for them to migrate. As an aside, the developer (Kim Griggs) mentioned that it took four months from proof-of-concept to production. I find this an amazing attribute of Ruby of Rails. The Umlaut took slightly less than six months. I should add these were both one person development teams.

Next up, David Walker talked about the new Worldcat API. What I found interesting about his talk is that he said his initial interest in the API was to be able to build ad-hoc union catalogs when he needed to. This was my original reason for trying out the Talis Platform when I was at Georgia Tech (although I could never figure out how to manage getting all the libraries I wanted to work with to get their holdings in a Platform store), so it definitely resonated. With the gravitational pull that OCLC already has in the U.S., this idea of ‘union catalog for specific problem set’ could really flourish. Of course, since not every library can afford to put their holdings in Worldcat (and therefore have access to the API) it may not be all that useful to the sorts of libraries that would gain the largest benefit.

While I was impressed by Winona Salesky’s and Michael Park’s presentation on XForms (there were definitely some slick features there!), my takeaway was that this technology seemed too difficult to deploy (when your options are browser extensions or Java servlets, I’ll pass on that). Am I missing something here? Java purists, you are free to mock me and my scripting language ways.

Interestingly, the Zotero presentation removed pretty much all references to RDF and the Semantic Web that appeared in their proposal.

The breakout session I attended on Tuesday was proposed by Jonathan Rochkind (who I co-presented with on the Umlaut) about finding a way to isolate only the open access content in OAIster. While maintaining a separate index of this data might be useful (sort of like what IndexData does, but they also include the non-free material; I am also harvesting some of this data with the intention of putting it into a platform store… but whoa does the clean-up of the data take time…), the group eventually decided that a web service that identified whether or not OAIster results were OA or not (via whitelists, blacklists, etc.) might be an easier first step. I’ve created a Google Group to carry this discussion forward here, if anybody is interested in participating.

I can’t remember which lightning talks happened on day one (is there a list published anywhere?), but it doesn’t matter. The day two lightning talks showed why having this style of presentation is so great. The reasons can be summed up by pictures of underpants and player piano midi files being played from musical scores generated from OCR. When the video files are up, definitely check those two out.

Did I mention that there were free drinks from 5:30-7:30 every day?

Karen Coyle opened day two with her keynote on RDA. Corey Harper closed the formal presentations the next day on the same topic. It was pretty neat how they managed to not step on each other’s toes. I think their message resounded pretty loudly: people like Code4libbers need to get involved in RDA and DCAM to ground it in reality. Ok, I’ll see what I can do.

LibLime’s MARC editor was quite nice. Granted, I’ve never used OCLC Connexion, but I would have to think something like this would be a strong competitor.

I’ll skip over Aaron Swartz’s presentation (based on the number of Flickr photos of his presentation, I figure there must be plenty about it already), but I plan on writing a bit about ThingDB and other document-centric databases really soon.

Skipping up to the DLF ILS API, both Emily and Terry’s presentation as well as the breakout session on this were incredibly useful.  I feel like the DLF has a pretty pragmatic approach to working on the problems of interoperability and Emily and Terry were really good emissaries to explain their goals to Code4lib.  During the breakout I introduced Jangle and later gave a lightning talk on it.  Since the cat’s out of the bag, I absolutely will have a post about that this week.  While I don’t exactly trust the vendors’ response to the DLF proposal (they meet on Thursday), I think that the potential of this is mutually beneficial to every party.  Jangle was quite well received, by the way.

Omeka seems exactly like what Greenstone promised to be, but wasn’t quite exactly (at least, in my mind).  I really like the focus and polish of this project.

Day three is where things started slipping for me.  Not only was my brain quite full at that point, but Che’s stomach virus apparently had taken a grip on me (and I apologize to anybody I paid that forward to).  I did manage to get out to see my family that night, though (who may not be as thankful to have seen me if I spread it to them, too).

This is a fun conference and I’m really proud to be a part of it.  Thankfully, Oregon State had everything so well organized that things still hummed along fine despite losing our ringleader.  I had a good time MCing, but we lost all of the dignity and professionalism when Jeremy had to leave.  “Slackerly” and “clownish” are more apt for me.  I know Roy Tennant and, sir, I am no Roy Tennant.