Some members of Boston.rb had a great discussion at last night’s hackfest about library dependencies. In particular, we were discussing the problem of libraries (gems) that depend on other libraries and how Rubygems and Bundler both promote implementation dependency instead of interface dependency. “Interfaces? We don’t need no stinkin’ interfaces!” I hear you cry. It’s true: Ruby doesn’t have interfaces in the same sense Java does, but it certainly has them in the object-oriented sense. We generally call them APIs.
Take the following scenario: you’re building a web application that you will deploy in a Java container, so you’re running on JRuby. You want to integrate with Twitter, so you add the twitter gem, which in turn depends on the yajl-ruby gem for JSON parsing. Sadly, yajl-ruby is C bindings for JSON. What you want to use is json-jruby, but you would have to fork the twitter gem to make it use that. It’s not just that yajl-ruby and json-jruby have different APIs — indeed, they don’t differ by much — it’s that the dependencies are declared in the .gemspec files, so even monkey-patching can’t save the day.
This is precisely why Eric Gamma (one of the Gang of Four) says you should, “program to an interface, not an implementation.” (See this interview.)
If twitter depended on an interface instead of an implementation, you would be free to use any implementation you wanted so long as it abided by the interface. But how, exactly, do you write an interface in Ruby?
provides
In a recent post, Marcin Kulic discusses this and other dependency problems in the Ruby/Gems landscape. His recommendation is that we add #provides to Gem::Specification and use it like so:
Gem::Specification.new do |gem|
gem.name = "my-json-implementation"
gem.version "2.7.4"
gem.provides "json", :version => "<= 1.2.0"
end
This is a fine idea in theory, but Nick Quaranto pointed out that modifying Rubygems is exceedingly difficult and it raises backwards-compatibility problems. Additionally, there are naming issues, since the first implementation of a set of functionality (JSON, YAML, LDAP, Twitter API) usually takes the most obvious name for the interface.
Michael Bleigh, in writing OmniAuth, a Rack-based authentication provider, found he needed JSON parsing, but he didn’t want to depend on a specific implementation precisely because he didn’t want to dictate what the clients of his gem should use. To scratch this itch, he created multi_json, which uses the Bridge Pattern. multi_json declares only two methods: MultiJson#encode(object) and MultiJson#decode(json_string). It has a number of “engines” that it can use. If you want to write a JRuby-based web application that uses OmniAuth, simply declare your dependency on json-jruby and then set the engine appropriately.
For functionality that already exists in several implementations — JSON encoding and decoding, LDAP access, &c. — the mission of an interface (either the “provides” version or the bridge gem version) is clear: extract out the most salient pieces of all the popular implementations and give them a common API. For functionality that does not yet exist — for example, the client API for a new web service — the interface builder is faced with a difficult decision: define the interface first or extract it later?
Pre-defining the interface has one key advantage: clients of the interface can easily depend on the interface rather than an implementation, so there won’t be a mad rush to change all the gems in the wild later. It also has a disadvantage: it reeks of premature decision making. Waiting to define the interface until several competing implementations have sprung up allows the interface writer to select only the most important aspects of the libraries.
Of course, in the real world, it is likely to be a mix. Perhaps you start with an implementation, try it out on a few projects, build a first version of an interface, and iterate on both. Interfaces, after all, can be versioned just like implementations. Of course, we hope they don’t change too often.
Above all, semantic versioning is key to building a great interface. Included in semantic versioning is proper use of deprecation. Any method, class, or module that is to be removed must first be deprecated in a minor or patch version and can only be removed in a major version.
Secondly, interface writers should take input from the community on the API. Perhaps the easiest way to do this is to inspect the popular implementations in a category on Ruby Toolbox for commonalities.
And, of course, only build interfaces where they’re truly necessary. Out of the thousands and thousands of gems in the wild, very few provide such core functionality that they are used by libraries that are in turn used by applications. Let’s keep in mind the problem we’re trying to solve and not go interface-crazy.
What does it mean to make an interface gem? multi_json is certainly a good start at a JSON interface. It provides core functionality (just two methods, #encode and #decode) and makes it easy for the client to plug in new engines. The Boston.rb group decided that a truly great interface gem would have at least one additional feature, though: an easy way for implementers to test the compliance of their implementations. This will vary in practice, but it might be an Rspec macro:
require 'multi_json/spec_macros'
describe MyJSONImplementation do
subject do
MyJSONImplementation
end
it_should_implement_json(:version => 1.0)
end
Or it could be a Module that gets mixed in to a test class:
require 'multi_json/test_support'
class MyJSONTest < Test::Unit::TestCase
include MultiJson::TestCase::VersionOnePointOh
def subject
MyJSONImplementation
end
end
In either case, the important point is that implementers have a dead-simple way to add a whole suite of compliance tests. One thing we discussed at the hackfest was creating an rspec_rfc gem that supports SHOULD, SHOULD NOT, MUST, and MUST NOT language so implementers could distinguish non-compliance (fails at least one MUST or MUST NOT), conditional compliance (passes all MUSTs and MUST NOTs, but fails at least one SHOULD or SHOULD NOT), and compliance (passes all). The problem here is that Rspec already uses should to mean what RFCs call MUST.
The other improvement that should be made to multi_json, and is important for all interfaces, is very thorough API documentation. multi_json only defines two methods, but the API contains more than method names. Parameter types, return types, possible exceptions, yielded block, yield params, and yield return values are all important. The current documentation for #encode and #decode looks like this:
# Decode a JSON string into Ruby.
#
# <b>Options</b>
#
# <tt>:symbolize_keys</tt> :: If true,
# will use symbols instead of strings
# for the keys.
def decode(string, options = {})
...
# Encodes a Ruby object as JSON.
def encode(object)
...
It should probably look like the following (and, yes, I will be submitting a patch):
# Decode a JSON String into a Ruby Object.
#
# @param [String] string the object in JSON representation
# @param [Hash, nil] options additional options
# @option options [true, false] :symbolize_keys if true,
# will use Symbols instead of Strings for Hash keys
# @return [Object] a Ruby Object
# @since 0.0.1
def decode(string, options = {})
...
# Encodes a Ruby Object as a JSON String
#
# @param [Hash, Array, Numeric, String, Symbol, #to_json] object a
# Ruby Object to be converted to JSON. Each object in
# the graph must be a Hash, Array, Numeric, String, or Symbol
# or declare a #to_json instance method.
# @raise [NoMethodError] if any object in the graph cannot be
# translated into JSON.
# @return [String] the Ruby Object as JSON
# @since 0.0.1
def encode(object)
...
Good candidates for interfaces tend to meet the following:
The three categories that come most easily to mind are formats (JSON, YAML, Markdown), protocols (LDAP, OpenID, HMAC, AES), and API clients (Twitter, Flickr, Delicious).
(Later: not ten minutes after posting this, I came across Yehuda Katz’s Moneta, an interface for key/value stores. Like multi_json, moneta is a great example of a bridge gem, though it could use some of the same improvements that multi_json could, namely providing implementers a test suite and having thorough documentation on the API.)
One criticism of interfaces is that they stifle innovation. It’s harder for me to write a truly new way of using LDAP if I have to comply with an interface. This is certainly a valid concern. The mitigation strategy should be extreme care in writing and curating interfaces. Don’t write one unless there’s an obvious need, and when you do write one, require only as much as you truly need for basic functionality. People can always depend on a specific implementation if they need more than the basics.
The Ruby community has a wealth of tools to measure code:
The underlying behavior that these tools measure tend to lie in power-law distributions. Perhaps 80% of your classes have 100% (or near-100%) code coverage, but 20% have less than 50%. Or maybe the cyclomatic complexity of 80% of your methods is less than 5, but 20% of them have a complexity greater than 16. The problem is that describing the mean or median of a power-law distribution is nearly meaningless. The reason is that the small number of instances at the head of the distribution share very little in common with the many that compose the tail.
The problem, thus, is not that we’re measuring the wrong things, but that we’re reporting the wrong measurements. If you look at a graph of mean-code-complexity over time and see that it is slowly but steadily getting worse (increasing), you can’t tell whether the bulk of your code is getting worse or whether there’s one particular module that is accruing all the technical debt. In the former case, you almost certainly have a communication or process problem; in the latter case, you might be doing everything perfectly and simply have to plan for a future refactoring.
Instead, power-law distributions are better described by the mean and/or median of the “head” (generally 20%) and “tail” (generally 80%) groups. Thus, each metric should have three configuration options:
bulk_delimiter: the dividing line between the “head” and “tail” groups. Defaults to 20%.head_min, head_max, or head_range for the ideal values for the “head” group. Default depends on the metric in question, but might be <= 50 for Flog.tail_min, tail_max, or tail_range for the ideal values for the “tail” group. Default depends on the metric in question, but might be <= 10 for Flog.What we have to change: the reporting of metrics we’re collecting. MetricFu’s graphs should have two lines for each metric. RCov’s coverage reports should have a divider between the “head” and “tail” groups and summary information for each.
What we don’t have to change: the collection methods. The hard part of these metrics libraries is the instrumentation of code and collection of data. Fortunately, we don’t have to change any of that. (NB: I haven’t actually read through all of the code for the above projects, so this isn’t actually a guarantee. I can, however, be fairly certain that we can keep at least 80% of the code :) )
A beautifully-laid out piece of analysis from Steven Pinker’s The Stuff of Thought on how the brain models stuff:
At first glance, the conceptual distinction between an object and a substance seems to be captured in the linguistic distinction between a count noun and a mass noun. Count nouns, like “apple” and “pebble,” tend to be used for bounded hunks of matter. Mass nouns, like “applesauce” and “gravel,” tend to be used for substances without their own boundaries. The two types of nouns are sharply distinguished by the grammar of English. We can enumerate and pluralize count nouns, but not mass nouns. When we refer to quantities, we have to use different quantifying words: “a pebble” is fine, but “a gravel” is not. We talk about “many pebbles” but not “many gravel,” and we talk about “much gravel” but not “much pebble” or “much pebbles.” And mass nouns can appear in public naked: “gravel is expensive,” “I like gravel.” Whereas count nouns generally cannot: “pebble is expensive,” “I like pebble.” An important clue to the mental model behind mass nouns is that, in some ways, they act like plurals of count nouns. They share some of their quantifiers: “more applesauce,” “more pebbles.” Or their ability to appear naked in a sentence: “I like applesauce,” “I like pebbles.” And their ability to appear with spatial words like “all over,” as in “applesauce was all over the floor,” and “pebbles were all over the floor.”
The overlap in grammar reflects a similarity in the way we conceive of substances. The things typically labelled with mass nouns and multitudes, the things typically labelled with plurals, which, together, may be called “aggregates.” Substances and multitudes both lack intrinsic boundaries and can spill out into any shape. They can coalesce: put some pebbles together with some pebbles and you still get pebbles, put some applesauce together with some applesauce and you still get applesauce. And they can be divided. None of this is true of the typical referent of a count noun, like a horse. No one is in doubt as to the boundary where a horse leaves off, and when you cut a horse in half, the result is not a horse.
Where a plural differs from a mass noun is that it is conceived as a set of individuals which can be identified and counted. A singular count noun like “pebble” stands for something that is bounded — delineated by a fixed shape, and not made up of individuals. A plural, like “pebbles,” stands for something that is unbounded and made up of individuals. A mass noun, like “applesauce,” stands for something that is neither bounded nor made up of individuals. All this suggests that our basic concepts about matter are not the concepts “count” and “mass,” but the mini-concepts, “bounded,” and “made up of individuals.” If so, we should see a fourth possibility: things that are both bounded and composed of individuals. We do, indeed. These are collective nouns, such as “committee,” “bouquet,” and “rock band.”
It’s not just nouns that care about boundedness and individuals. Verbs do, too. As we saw earlier, verbs like “pour” require aggregates, like “water” or “pebbles.” Verbs like “smear” and “streak” apply to substances. And verbs like “scatter” and “collect” apply to multitudes. This is because the concept of an action depends on the number and kind of things it affects — as in the difference between “eat” and “drink,” “throw” and “scatter,” “murder” and “massacre.”
The intuitive materials science behind the count/mass distinction assumes a Play-Dough world in which objects are molded out of the substance. Rocks are made of rock, glasses are made of glass, beers are made of beer, cats are made of cat. The model breaks down when an object can’t be construed as having been formed from a scoop of material. A television isn’t made out of something called “television,” so we can’t say that, “a steamroller left television all over the road.” The distinction also breaks down when we put a substance under a powerful enough microscope. We use the word “rice” to refer to a cup of it, a grain of it, or even a fragment of a grain of it, but as we zoom closer and closer, we reach a point at which we aren’t seeing rice any more. Perhaps if humans could see the crystals, fibers, cells, and atoms making up matter, we would never have developed a count/mass distinction in the first place.
The count/mass distinction in our minds is not just unfettered by the object/substance distinction in the world. It is unfettered by the physical world altogether. It is best thought of as cognitive lens or attitude by which the mind can construe almost anything as a bounded, countable item, or as a boundryless, continuous medium. We see this in a distinctive kind of mass noun that does what count nouns usually do. These are mass hypernyms — superordinates, such as “furniture,” “fruit,” “clothing,” “mail,” “toast,” and “cutlery.” Though they don’t refer to a substance — chairs and tables aren’t made out of “furniture” — the words can’t refer directly to the individual objects they stand for, either. They require a special classifier noun, as in “a stick of furniture,” “an article of clothing,” or the general-purpose classifier “piece.”