Just because a project is popular does not necessarily mean that it is successful.  It just means at that point in time and duration it is also getting a lot of attention.   I’ve been involved with a wide variety of open source projects over the last 15 years.  Several I’ve started. Several I’ve participated in from both as a community member and as a committer.   Some I would say have been popular at one point or another, other’s I would say never hit the “High School” popularity meter thresh hold.

Popularity is defined by Merriam-Webster’s Learner’s dictionary as, “state of being liked, enjoyed, accepted, or done by a large number of people : the quality or state of being popular”.

Successful is defined as “having the correct or desired result.”   So being successful does not necessarily equal popularity.   A successful open source project is one that has met its desired goals and objectives.   One of them could be, to become popular, and blogged about, and tweeted about constantly, but the winds of popularity change at a moments notice.  Success lasts for as long as the project is meeting its stated goals.

Success is defined by the commiters and the community that forms around the project.  Is it meeting its user’s needs?  Is it growing its user community and committer base?  Is it adapting and changing to meet knew requirements?   These are just some of the criteria that a project can be measured on to be successful.  Yes, popularity can be one criteria, but it should not be the sole deciding factor.

We need to be careful on tying success to popularity alone.   There are many projects out there that I would consider successful that have never hit the popularity threshold.  Those projects may not be used by Millions of people, but they have built a community around their project, and provide value to that community.   How many people are using it, or number of companies that have adopted it, should not take away from the success of the project if it has met its objectives.

So success is going to mean different things to different people.  There is no one way to measure if project is successful.  To me, if your project even gets a handful of people forming a community around it, you are already successful.

If I leave said company’s ecosystem, and want my the same media elsewhere, if they do not give me a way to download those purchases and then upload them somewhere else, then I’m screwed as a consumer.  I’ll have to repurchase said items again.   The other disturbing item is that the book formats are not standardized, or they have DRM added to them.   Copyright should be protected but not at the expense of allowing a consumer to legaly move their digital content to other platforms.

Apple isn’t the sole cause of concern.  As more content moves to the cloud for storage, we as consumers should be demanding that our content is stored and available in a standard format for offline consumption.   Amazon Kindle and the Barnes and Noble Nook are also cause for concern, but not as much because you aren’t locked into a particular device to view the content.  Amazon and Nook should make the content available that you purchase in a standard format so that you do not need to re-purchase it elsewhere.

While I’m not a big fan of the Google book store, at least they provide some instructions on how to download your content for offline viewing, into ePub or PDF format.   Personally, ePub is a much better format than PDF and works better across devices.   Like or hate, Google, they are at least giving you the option to take your purchases with you if you decide to leave their cloud based system.

The big problem though with the Apple announcement is the fact that it locks you into the iPad/iPhone/iCloud ecosystem.  If I was part of a schools IT system, I’d be raising all sorts of hell about this lock in.  For one thing, it requires an iPad.  Not something that every school can or should have to purchase to educate children.   If the content was available in a standardized format, and available on multiple non-Apple devices I probably would not have as much problem with the announcement.   The costs are also out of wack for the public education system, as the books have to be purchased each year by the students.  Currently most schools loan the books to the students, and they turn them back in at the end of the year.   The current format of making the students by each of the books, is cost prohibitive to many in the public and even private education systems.

The proprietary nature can be addressed through standardization, and making sure we as consumers are demanding that our digital content work across multiple devices.  If you buy a book through the bookstores for Amazon, Apple, Google, Barnes and Noble, or other retailers, that purchase should work on whatever device you have.   If you want to use it on your Kindle, and purchased it from Apple, it should work.

Unfortunately, we as consumers seem to be getting complacent.  We seem to be taking the easy way out, and only really are protactive on something once it really seems to affect us.  Take a look at the recent Blackout by Wikipedia on the whole SOPA thing.   The vast majority of people probably had no clue what was happening until they could no longer access their content.   Now imagine the same thing happening for your other digital content being held in the Cloud.

I love the idea of digital textbooks, especially for the higher education institutions, I just think it needs to be portable and not locked into proprietary clouds and data formats.  Knowledge should not be caged.

The new project I’ll be working on gets me back to some of the things I like to do, plus will finally get me to learn some of the newer languages and play with  other technologies.   It should also free me up a bit to address some of the open source projects I’ve had to neglect over the last year at eclipse.  So hopefully, I’ll be able to get to some of the eclipse XSL Tools bugs reports, and maybe..just maybe, I’ll look at Orion and see what it would take to get an XML/XSLT editor going there.

I’ve got a lot of little projects going on at github as well, and I’ve got some ideas around improving the state of  web service documentation that I feel are necessary to explore.   The early part of this year I have some presentations lined up at STAR and HR-XML around REST and Web 2.0.  So I hope to explore how to get organisations like OAGI and related data standard bodies looking at not just XML but how to play well with JSON in the new Web 2.0 world.

So I wish the Turmeric SOA project well, but I’m really looking forward to getting my hands back into some new stuff this year.  Embrace change, good things tend to happen.

We will use the jetty:run-war plugin.   To get started create a maven war project.  You can use the maven webapp archetype to create the project.  Once that is completed, you will need to add a dependency to your Turmeric SOA service implementation project and add a plugin entry to specify the version of jetty to use.   You will end up with something like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example.test</groupId>
  <artifactId>test-hello-world</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <name>Test Hello World Web App</name>
  <packaging>war</packaging>

  <dependencies>
     <dependency>
         <groupId>org.ebayopensource.turmeric.impl</groupId>
         <artifactId>DemoHelloWorldV1</artifactId>
         <version>1.0.0</version>
     </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <version>7.5.4.v20111024</version>
      </plugin>
    </plugins>
  </build>

</project>

To test and deploy your web service, you just need to type run: mvn jetty:run-war

You can test that the service is deployed by point your favorite web browser to http://localhost:8080/DemoHelloWorldV1?wsdl

Remember to replace the above with the name of your service.

I typically use this for some quick testing of changes done to a service, or items that need some sort of integration testing. You could also use this to help with some automated testing, but in those cases I recommend writing your test to extend from the Turmeric SOA jetty-common-tests framework. This framework provides an embedded instance of jetty that can be used in integration tests.

https://www.ebayopensource.org/index.php/Turmeric/Downloads

Included in this release are several new features and improvements.

• Cassandra storage support for both Rater Limiter, Runtime, and Monitoring Console. Metrics are captured and stored in a cassandra data store. The JPA option for relational databases is still available for Monitoring and Runtime.
• Jetty Turmeric application server. A pre-configured version of jetty 7.4 with turmeric runtime components and common service handlers has been provided for download. This should make testing and development of turmeric services simplier. All runtime items are deployed to a common location. Support for JPA (using Derby) and Cassandra data sources are available, as well as access to the various Security handlers.
• Protobuf support has been added as an option for serialization and deserialization for services to use.
• Expanded support for RESTful web services, using GET, PUT, POST, and DELETE operations.
• OSGI improvements with the way classloaders are implemented. This will allow the turmeric runtime components to be used within an OSGI container.
• Additional work on the Repository Service has been done.  This version requires WSO2 Governance Registry 4.x, and the service is still beta.  Work continues to implement the full functionality against the WSO2 Governance Registry.
• Policy Admin UI is now released as 1.0.0.  Minor bug fixes have been included.  See the release notes for more information.

Many bug fixes, and tweaks have been done as well. For further information please seen the release notes.

Natively it doesn’t support JSONP, which is used by many javascript applications.   However, you can easily add this support to your web service by adding a filter in your web.xml for your service.   Hidden out on Google Code is a project called, jsonp-java.   It basically looks for response types that have the application/json, text/json, application/x-json, and other related mime-types.   If there is a callback parameter specified in the request, it will wrap the JSON data returned with the function name to be called.

To add this support to a Turmeric SOA web service or any web service that returns json, add the following code to your web.xml.

  <filter>
      <display-name>jsonp</display-name>
      <filter-name>jsonp</filter-name>
      <filter-class>org.jsonp.JsonpFilter</filter-class>
      <init-param>
         <param-name>jsonp</param-name>
         <param-value>jsonpCallback</param-value>
      </init-param>
      <init-param>
         <param-name>json-mime-types</param-name>
         <param-value>application/json</param-value>
      </init-param>
   </filter>

   <filter-mapping>
      <filter-name>jsonp</filter-name>
      <url-pattern>*</url-pattern>
   </filter-mapping>


   <mime-mapping>
      <extension>json</extension>
      <mime-type>application/json</mime-type>
   </mime-mapping>

If you need to support other mime types, add an appropriate mime mapping to the web.xml. If you are using maven to build your WAR files, then you will need to add the jsonp.jar as a dependency during your war creation.

      <dependency>
         <groupId>org.jsonp</groupId>
         <artifactId>jsonp</artifactId>
         <version>1.0.1</version>
      </dependency>

Unfortunately the jsonp jar file is not in maven central so you will need to checkout the code and install it into your local repository. Just run a mvn clean install after checking out the project. The code currently is only available in a Subversion repository, but I’ll put my git version of the repository online shortly at github.

Once everything is setup and deployed. To have a JSON request return as a JSONP request with Turmeric, you can add a jsonpCallback request parameter as part of your URL request.

localhost:8083/starws/StarWebServiceV1?X-TURMERIC-OPERATION-NAME=pullMessage&X-TURMERIC-RESPONSE-DATA-FORMAT=JSON&jsonpCallback=someFunction

Set the jsonpCallback query string to the name of the function that should be executed. When the response is returned, the JSON data will be wrapped with the function to be executed on the callback.

someFunction({
    "jsonns.ns2": "http://www.starstandards.org/webservices/2009/transport",
    "jsonns.ns3": "http://www.starstandards.org/webservices/2009/transport/bindings",
    "jsonns.ms": "http://www.ebayopensource.org/turmeric/common/v1/types",
    "jsonns.xs": "http://www.w3.org/2001/XMLSchema",
    "jsonns.xsi": "http://www.w3.org/2001/XMLSchema-instance",
    "ns2.PullMessageResponse": {
        "ns2.payload": {
            "ns2.content": [
                {
                    "@id": "1"
                }
            ]
        }
    }
});

The reason for the funky JSON, is that Turmeric SOA supports it’s own flavor of namespaces for JSON. Useful for dealing with existing XML formats that also need to served in a JSON format. There are multiple ways to handle this and the above representation is just another flavor. More on this in a future post.

The jsonp filter can be used with any web service that can return JSON data, it just needs to have the jsonpCallback query parameter set, so this isn’t limited to the Turmeric SOA framework.

With the popularlity of JSON and other NoSQL data standard formats, the complexity and in some cases the plain verbosity of XML formats are being shunned.  However, XML and the abilities and standards that have formed around it have become key to the enterprise and their business processes.   However, the needs of their customers require that they start to support multiple formats.    To this end, tooling frameworks like Axis2 have started to add support for taking WSDL based services and generate JSON responses.

Enterprises need to live in this post SOAP world, but leverage the expertise and SOA infrastructures they have developed over the years.   Axis2 is one way to do it, but it doesn’t provide monitoring and policy support of the box.   Another alternative is the Turmeric SOA project from ebay.    Natively out of the box one can take a WSDL like, the one provided by the STAR standard organization, and add support not only from SOAP 1.1/1.2, but also for REST style services serving XML, JSON, and other any other data format you would need to support.

There is a catch though, Turmeric SOA was not designed with full SOAP and the W3C web service stack in mind. It uses WSDL to only describe the operations and the data formats supported by the service.   So advanced features like WS-Security, WS-Reliable Messaging, and XML Encryption are not natively built into Turmeric.   Depending on your needs you will need to work with pipeline Handlers and enhance the protocol processors to support some of the advance features.   However, these are items that can be worked around, and it can interoperate with existing web services to act as a proxy.

As an example, the STAR organisation provides a web service specification that has been implemented in the automative industry to provide transports for their specifications.  Using a framework like Turmeric SOA existing applications can be made available to trading partners and consumer of the service in multiple formats.   As an example, one could provide data in RESTful xml:

<?xml version='1.0' encoding='UTF-8'?>
<ns2:PullMessageResponse xmlns:ms="http://www.ebayopensource.org/turmeric/common/v1/types" xmlns:ns2="http://www.starstandards.org/webservices/2009/transport">
   <ns2:payload>
      <ns2:content id="1"/>
   </ns2:payload>
</ns2:PullMessageResponse>

Or one can provide the same XML represented in a JSON format:

{
    "jsonns.ns2": "http://www.starstandards.org/webservices/2009/transport",
    "jsonns.ns3": "http://www.starstandards.org/webservices/2009/transport/bindings",
    "jsonns.ms": "http://www.ebayopensource.org/turmeric/common/v1/types",
    "jsonns.xs": "http://www.w3.org/2001/XMLSchema",
    "jsonns.xsi": "http://www.w3.org/2001/XMLSchema-instance",
    "ns2.PullMessageResponse": {
        "ns2.payload": {
            "ns2.content": [
                {
                    "@id": "1"
                }
            ]
        }
    }
}

The above is generated from the same web service, but with just a header changed to indicate the data format that should be returned. No actual changes to business logic or the web service implementation code itself has to change. In Turmeric this is handled with the Data Binding Framework and its corresponding pipeline handlers. With Axis2 this is a message request configuration entry. Regardless of how it is done, it is important to be able to leverage existing services, but provide the data in a format that your consumers require.

For those that are interested, I’ve created a sample STAR Web Service that can be used with Turmeric SOA to see how this works. Code is available at github.

While Turmeric handles the basics of the SOAP protocol, advance items like using and storing information in the soap:header are not as easily supported. You can get to the information, but because the use of WSDL in Turmeric Services are there just to describe the data formats and messages, the underlying soap transport support is not necessarily leveraged to the full specification. Depending on your requirements, Axis2 may be better, but Turmeric SOA provides additional items like Service Metrics, Monitoring Console, Policy Administration, Rate Limiting through XACML 2.0 based policies, and much more. If you already have existing web services written the W3C way, but need to provide data in other formats, Turmeric can be used along side. It isn’t a one or the other proposition. Leverage the tools that provide you the greatest flexibility to provide the data to your consumers, with the least amount of effort.

It’s that time of year again…EclipseCon North America submission time.

Like every year, the program committee is asking you to submit your talk early.  Ian and Doug have been cracking the whip pretty hard telling us to blog, so here I am.    One of the topics that came up recently is how do we choose the talks, what are we looking for, who are the rock stars that people want to see.   One sure way is to ask what you want to see or attend at the conference, so provide your feed back here, to Doug, Ian, or any of the other committee members.

Last year, I referenced a paper, “How to Get a Paper Accepted at OOPSLA“, in which it outlined some of the items that, that committee was looking for in a submission.  So I’ll jot down some things that I’m looking for below.

I don’t have a particular technology or niche that I focus on any more, it used to be XML and the related technologies around it, but in most cases the XML tools at eclipse are good enough for most people’s uses.   So I’m not necessarily looking in that area, unless something REALLY REALLY cool comes along.  Also if you submitted the same talk in the past or presented it in the past….not interested.

I’m more focused over the last year or two on quality of the code that is produced by a project and developers in general.   How to make sure we keep the code maintainable and the practices around that.   So static analysis tools like PMD, and FindBugs are interesting, but I want to know how your projects are using these tools to improve your code.   What are the new and interesting things coming out of these tools?  Projects like Code Recommenders peak my interest, and the additions they provide.   Mylyn…well they do interesting things, so I’m looking not so much the Core Mylyn task oriented project, but their sub projects, and how they are improving developers lives.

Experience reports from projects migrating to or from a centralized version control system to a distributed version control system.  The pains, the gotchas.    How can EGit be improved, how are you using JGit to get your work done.   Has moving to a distributed version control system made your life easier, harder?  Are they over hyped?

Builds…Builds…Builds.   The battle of the builds continues to happen, but more importantly what can we do about p2?   p2 started as provisioning system for an application, but is being used more and more for to provision a build and a repository for those build artifacts.   Are you doing any work in the enterprise or open source projects around addressing the issue of stale or non-existant p2 repositories.   How does this affect the reproducability of your build?  Are there ways to address this?  How does the p2 model of a repository compare the Maven Central concept?   Should the Eclipse foundation replicate the model for p2 and if so how?

What are the problems that a project and committers coming from a corporate environment experience when working on a previously closed project that is now open?   How did you adapt, what leasons can be taken away from this?

The web and mobile space is still hot, but how do we evolve our development environment and particularly the eclipse interface to the new touch screen mentality?   Is there something within Eclipse 4.x platform that allows for adapting to the new platform?  Is SWT even the right technology going forward?  Should we be looking at new ways to develop our user interfaces so they are cleaner and easier to use?   How do we get more UI specialists involved with open source projects in general?

In general, I’m interested in things that challenge the status quo, that make me think, and challenge my beliefs.   That help make me more productive, improve the quality of the code I produce, and make it easier for people that use that code.  Give me more demos, and less talk.  For tutorials, I really want lots of hands on, and short introductions to the topics that will be worked on in an exercise.   Get me up and running quickly.

So those are the types of things that will peak my interest on a submission.  It’s not so much the underlying technology (i.e. xtext, xml, emf, css, javascript, json, java, scala, osgi, etc), but what does it enable and improve upon.  Tell me how it improves or challanges the status quo.

I look forward to your many submissions.

On twitter I mentioned we had a situation like this on the Turmeric SOA project, and I’ve talked about similar situations regarding various eclipse project code as well.  For Turmeric there is a Repository Service, which provides a specification for Governance of Assets for a service as well as the lifecycle management of these assets.   The initial implementation of which for the WSO2 Governance Registry provider can be see here.   The code is done the way I see a majority of WSDL web service implementations done.  The methods contain all code related operations in the one class file.  This leads to several problems.   First the class is over 1500 lines long, making more difficult to understand what the class is doing and maintain.  Second it has more than one reason to change.   The class is handling Update, Delete, Subscription, Submittal, Add, etc of various repository assets.   So it is a magnet for any change that has to happen in the implementation.  It is difficult to test in isolation, and also tends to lead to some over use of static utility methods.  (The static utility method is another plague, but that is for another post).

There is a general rule of thumb that your classes should ideally be no more 500 lines long, and that they should have very few reasons to change.   The implementation above was refactored into the following:

1. Provider Implementation.
2. Operation Implementations.

Each of the operations now only has a single reason to change.  Each class is less than 500 lines, and is a bit easier to understand.   Just because an Interface defines all the methods does not mean that you have to put all the functionality within the class and method.  Break it up, make your code easier to test, and easier to maintain.     Ideally, I’d refactor this service into at least two or more separate services.   Are there other ways to handle this?  Is there another set of patterns besides Proxy/Delegate that could be used to handle this situation?

Stop by and learn about it’s capabilities, and how you can help out with the project.