Enterprise SOAP Services in a Web 2.0 REST/JSON World

The W3C Web Service StackMany enterprises have devoted a large amount of time and energy into supporting the W3C SOAP and Web Service stack.  In fact many people,  when the term Service Oriented Architecture is brought up automatically think this is SOAP and the Web Service stack that goes along with it.   However, this is wrong.   RESTful JSON/XML web services are also a key aspect to implementing a SOA.  SOA is much more though than just a web service a series of web services.  There is the need to monitor, track, and manage access to the services.    So regardless if you are implementing SOAP based web services or not the concepts developed over the years still apply.

With the popularlity of JSON and other NoSQL data standard formats, the complexity and in some cases the plain verbosity of XML formats are being shunned.  However, XML and the abilities and standards that have formed around it have become key to the enterprise and their business processes.   However, the needs of their customers require that they start to support multiple formats.    To this end, tooling frameworks like Axis2 have started to add support for taking WSDL based services and generate JSON responses.

Enterprises need to live in this post SOAP world, but leverage the expertise and SOA infrastructures they have developed over the years.   Axis2 is one way to do it, but it doesn’t provide monitoring and policy support of the box.   Another alternative is the Turmeric SOA project from ebay.    Natively out of the box one can take a WSDL like, the one provided by the STAR standard organization, and add support not only from SOAP 1.1/1.2, but also for REST style services serving XML, JSON, and other any other data format you would need to support.

There is a catch though, Turmeric SOA was not designed with full SOAP and the W3C web service stack in mind. It uses WSDL to only describe the operations and the data formats supported by the service.   So advanced features like WS-Security, WS-Reliable Messaging, and XML Encryption are not natively built into Turmeric.   Depending on your needs you will need to work with pipeline Handlers and enhance the protocol processors to support some of the advance features.   However, these are items that can be worked around, and it can interoperate with existing web services to act as a proxy.

As an example, the STAR organisation provides a web service specification that has been implemented in the automative industry to provide transports for their specifications.  Using a framework like Turmeric SOA existing applications can be made available to trading partners and consumer of the service in multiple formats.   As an example, one could provide data in RESTful xml:


<?xml version='1.0' encoding='UTF-8'?>
<ns2:PullMessageResponse xmlns:ms="http://www.ebayopensource.org/turmeric/common/v1/types" xmlns:ns2="http://www.starstandards.org/webservices/2009/transport">
   <ns2:payload>
      <ns2:content id="1"/>
   </ns2:payload>
</ns2:PullMessageResponse>

Or one can provide the same XML represented in a JSON format:

{
    "jsonns.ns2": "http://www.starstandards.org/webservices/2009/transport",
    "jsonns.ns3": "http://www.starstandards.org/webservices/2009/transport/bindings",
    "jsonns.ms": "http://www.ebayopensource.org/turmeric/common/v1/types",
    "jsonns.xs": "http://www.w3.org/2001/XMLSchema",
    "jsonns.xsi": "http://www.w3.org/2001/XMLSchema-instance",
    "ns2.PullMessageResponse": {
        "ns2.payload": {
            "ns2.content": [
                {
                    "@id": "1"
                }
            ]
        }
    }
}

The above is generated from the same web service, but with just a header changed to indicate the data format that should be returned. No actual changes to business logic or the web service implementation code itself has to change. In Turmeric this is handled with the Data Binding Framework and its corresponding pipeline handlers. With Axis2 this is a message request configuration entry. Regardless of how it is done, it is important to be able to leverage existing services, but provide the data in a format that your consumers require.

For those that are interested, I’ve created a sample STAR Web Service that can be used with Turmeric SOA to see how this works. Code is available at github.

While Turmeric handles the basics of the SOAP protocol, advance items like using and storing information in the soap:header are not as easily supported. You can get to the information, but because the use of WSDL in Turmeric Services are there just to describe the data formats and messages, the underlying soap transport support is not necessarily leveraged to the full specification. Depending on your requirements, Axis2 may be better, but Turmeric SOA provides additional items like Service Metrics, Monitoring Console, Policy Administration, Rate Limiting through XACML 2.0 based policies, and much more. If you already have existing web services written the W3C way, but need to provide data in other formats, Turmeric can be used along side. It isn’t a one or the other proposition. Leverage the tools that provide you the greatest flexibility to provide the data to your consumers, with the least amount of effort.

Posted in json, open source, turmeric, web services, wsdl | Leave a comment

Repost: EclipseCon 2012 – What I WANT!

I’m reposting this, after having reviewed the first batch of submissions.  I’ve gone through and higlighted some key sections, at least for me.  You have through Monday for early bird selection.  Final submission deadline is Nov 18th.

It’s that time of year again…EclipseCon North America submission time.

Like every year, the program committee is asking you to submit your talk early.  Ian and Doug have been cracking the whip pretty hard telling us to blog, so here I am.    One of the topics that came up recently is how do we choose the talks, what are we looking for, who are the rock stars that people want to see.   One sure way is to ask what you want to see or attend at the conference, so provide your feed back here, to Doug, Ian, or any of the other committee members.

Last year, I referenced a paper, “How to Get a Paper Accepted at OOPSLA“, in which it outlined some of the items that, that committee was looking for in a submission.  So I’ll jot down some things that I’m looking for below.

I don’t have a particular technology or niche that I focus on any more, it used to be XML and the related technologies around it, but in most cases the XML tools at eclipse are good enough for most people’s uses.   So I’m not necessarily looking in that area, unless something REALLY REALLY cool comes along.  Also if you submitted the same talk in the past or presented it in the past….not interested.

I’m more focused over the last year or two on quality of the code that is produced by a project and developers in general.   How to make sure we keep the code maintainable and the practices around that.   So static analysis tools like PMD, and FindBugs are interesting, but I want to know how your projects are using these tools to improve your code.   What are the new and interesting things coming out of these tools?  Projects like Code Recommenders peak my interest, and the additions they provide.   Mylyn…well they do interesting things, so I’m looking not so much the Core Mylyn task oriented project, but their sub projects, and how they are improving developers lives.

Experience reports from projects migrating to or from a centralized version control system to a distributed version control system.  The pains, the gotchas.    How can EGit be improved, how are you using JGit to get your work done.   Has moving to a distributed version control system made your life easier, harder?  Are they over hyped?

Builds…Builds…Builds.   The battle of the builds continues to happen, but more importantly what can we do about p2?   p2 started as provisioning system for an application, but is being used more and more for to provision a build and a repository for those build artifacts.   Are you doing any work in the enterprise or open source projects around addressing the issue of stale or non-existant p2 repositories.   How does this affect the reproducability of your build?  Are there ways to address this?  How does the p2 model of a repository compare the Maven Central concept?   Should the Eclipse foundation replicate the model for p2 and if so how?

What are the problems that a project and committers coming from a corporate environment experience when working on a previously closed project that is now open?   How did you adapt, what leasons can be taken away from this?

The web and mobile space is still hot, but how do we evolve our development environment and particularly the eclipse interface to the new touch screen mentality?   Is there something within Eclipse 4.x platform that allows for adapting to the new platform?  Is SWT even the right technology going forward?  Should we be looking at new ways to develop our user interfaces so they are cleaner and easier to use?   How do we get more UI specialists involved with open source projects in general?

In general, I’m interested in things that challenge the status quo, that make me think, and challenge my beliefs.   That help make me more productive, improve the quality of the code I produce, and make it easier for people that use that code.  Give me more demos, and less talk.  For tutorials, I really want lots of hands on, and short introductions to the topics that will be worked on in an exercise.   Get me up and running quickly.

So those are the types of things that will peak my interest on a submission.  It’s not so much the underlying technology (i.e. xtext, xml, emf, css, javascript, json, java, scala, osgi, etc), but what does it enable and improve upon.  Tell me how it improves or challanges the status quo.

I look forward to your many submissions.

Posted in eclipse, eclipsecon | 1 Comment

Refactoring: Maintainable Web Services

One of the problems I have with WSDL based web services is the fact that operations are typically generated into one big interface file that must be implemented.   I’ve seen WSDLs with 50 to 70 operations, and this leads to a Service/Client Implementation that has 50 to 70 methods that need to be implemented.   The problem here is that the code becomes a magnet for change.  The correct thing to do is to refactor the WSDL into multiple services, but if you can’t do this and you don’t have control over the WSDL, how can you deal with this situation?

On twitter I mentioned we had a situation like this on the Turmeric SOA project, and I’ve talked about similar situations regarding various eclipse project code as well.  For Turmeric there is a Repository Service, which provides a specification for Governance of Assets for a service as well as the lifecycle management of these assets.   The initial implementation of which for the WSO2 Governance Registry provider can be see here.   The code is done the way I see a majority of WSDL web service implementations done.  The methods contain all code related operations in the one class file.  This leads to several problems.   First the class is over 1500 lines long, making more difficult to understand what the class is doing and maintain.  Second it has more than one reason to change.   The class is handling Update, Delete, Subscription, Submittal, Add, etc of various repository assets.   So it is a magnet for any change that has to happen in the implementation.  It is difficult to test in isolation, and also tends to lead to some over use of static utility methods.  (The static utility method is another plague, but that is for another post).

There is a general rule of thumb that your classes should ideally be no more 500 lines long, and that they should have very few reasons to change.   The implementation above was refactored into the following:

  1. Provider Implementation.
  2. Operation Implementations.

Each of the operations now only has a single reason to change.  Each class is less than 500 lines, and is a bit easier to understand.   Just because an Interface defines all the methods does not mean that you have to put all the functionality within the class and method.  Break it up, make your code easier to test, and easier to maintain.     Ideally, I’d refactor this service into at least two or more separate services.   Are there other ways to handle this?  Is there another set of patterns besides Proxy/Delegate that could be used to handle this situation?

Posted in clean code, craftsmanship, eclipse, refactoring, turmeric | 3 Comments

Turmeric SOA at JavaOne

On Monday October 3, JavaOne will hold their Birds of a Feather.   Turmeric SOA is on the card.   We’ll be introducing what the project includes, and how you can get involved.   Sastry Malladi from eBay, the lead Architect behind the platform will be there to answer your questions.  Turmeric SOA can be used complimentary to your existing SOA solution, and it can be used to help extend your existing services, so they can provide other data formats.

Stop by and learn about it’s capabilities, and how you can help out with the project.

Posted in open source, turmeric, web services, wsdl | Leave a comment

Re-Writing Eclipse Commit History

So you took the plunge and migrated your project from CVS/SVN to GIT?   However at the time you did so, you didn’t think about possibly splitting out the projects into multiple git repositories.   You and your project were still getting familiar with GIT, and didn’t know the advantages/benefits of splitting the repo up could bring.    This is the situation that occurred when the WTP Incubator project and its components made the move to git.

When we originally decided to make the switch we just took the easy way out and migrated the incubator directory in CVS to git.   All the components were arranged in the following directories:

sourceediting
    development
    features
    plugins
    examples
    documentation
    tests

webservices
    development
    features
    plugins
    examples
    documentation
    tests

The incubator project is made up of several incubating components, including Vex, XML Security, XQuery, RelaxNG, and others. You can view the current git repository here.

With Vex getting ready to graduate, and move to the Mylyn Docs project, we needed to split out the commit history just so that it represented the relevant items for the Vex project. To help with this the command line git filter-branch command was used to re-write the commit history to remove the other projects all together. The newly cleaned git repository was then pushed to a clean git repository on GitHub.

To clean the tree and remove all empty commits during the process the following bash script was used.


git filter-branch -f --prune-empty --tree-filter 'rm -rf webservices' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/examples' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/development/org.eclipse.wst.xml.relaxng.releng' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/development/org.eclipse.wst.xml.security.*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/development/org.eclipse.wst.xquery.*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/development/org.eclipse.wst.xquery.repository' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/features/org.eclipse.wst.xml.relaxng.*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/features/org.eclipse.wst.xml.security.*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/features/org.eclipse.wst.xquery*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/features/org.eclipse.wst.xquery.build.feature' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/features/org.eclipse.wst.marklogic*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/com.google.gson' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/com.matinprobst.xqpretty' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/com.thaiopensource.jing' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.antlr.runtime_v31' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.apache.xml.security' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.eclipse.wst.rng*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.eclipse.wst.xml.relaxng*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.eclipse.wst.xml.security*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.eclipse.wst.xquery*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/plugins/org.kohsuke.rngom' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/tests/org.eclipse.wst.xml.security*' HEAD
git filter-branch -f --prune-empty --tree-filter 'rm -rf sourceediting/tests/org.eclipse.wst.xquery*' HEAD

If you use the --all option it will rewrite it across all branches, otherwise, it runs against the current branch you have checked out.

Effectively once the filter-branch is done, you will effectively have purged all history of the prior components.

Posted in eclipse, git, open source, release engineering, vex | 1 Comment

EclipseCon 2012 – What I WANT!

It’s that time of year again…EclipseCon North America submission time.

Like every year, the program committee is asking you to submit your talk early.  Ian and Doug have been cracking the whip pretty hard telling us to blog, so here I am.    One of the topics that came up recently is how do we choose the talks, what are we looking for, who are the rock stars that people want to see.   One sure way is to ask what you want to see or attend at the conference, so provide your feed back here, to Doug, Ian, or any of the other committee members.

Last year, I referenced a paper, “How to Get a Paper Accepted at OOPSLA“, in which it outlined some of the items that, that committee was looking for in a submission.  So I’ll jot down some things that I’m looking for below.

I don’t have a particular technology or niche that I focus on any more, it used to be XML and the related technologies around it, but in most cases the XML tools at eclipse are good enough for most people’s uses.   So I’m not necessarily looking in that area, unless something REALLY REALLY cool comes along.  Also if you submitted the same talk in the past or presented it in the past….not interested.

I’m more focused over the last year or two on quality of the code that is produced by a project and developers in general.   How to make sure we keep the code maintainable and the practices around that.   So static analysis tools like PMD, and FindBugs are interesting, but I want to know how your projects are using these tools to improve your code.   What are the new and interesting things coming out of these tools?  Projects like Code Recommenders peak my interest, and the additions they provide.   Mylyn…well they do interesting things, so I’m looking not so much the Core Mylyn task oriented project, but their sub projects, and how they are improving developers lives.

Experience reports from projects migrating to or from a centralized version control system to a distributed version control system.  The pains, the gotchas.    How can EGit be improved, how are you using JGit to get your work done.   Has moving to a distributed version control system made your life easier, harder?  Are they over hyped?

Builds…Builds…Builds.   The battle of the builds continues to happen, but more importantly what can we do about p2?   p2 started as provisioning system for an application, but is being used more and more for to provision a build and a repository for those build artifacts.   Are you doing any work in the enterprise or open source projects around addressing the issue of stale or non-existant p2 repositories.   How does this affect the reproducability of your build?  Are there ways to address this?  How does the p2 model of a repository compare the Maven Central concept?   Should the Eclipse foundation replicate the model for p2 and if so how?

What are the problems that a project and committers coming from a corporate environment experience when working on a previously closed project that is now open?   How did you adapt, what leasons can be taken away from this?

The web and mobile space is still hot, but how do we evolve our development environment and particularly the eclipse interface to the new touch screen mentality?   Is there something within Eclipse 4.x platform that allows for adapting to the new platform?  Is SWT even the right technology going forward?  Should we be looking at new ways to develop our user interfaces so they are cleaner and easier to use?   How do we get more UI specialists involved with open source projects in general?

In general, I’m interested in things that challenge the status quo, that make me think, and challenge my beliefs.   That help make me more productive, improve the quality of the code I produce, and make it easier for people that use that code.  Give me more demos, and less talk.  For tutorials, I really want lots of hands on, and short introductions to the topics that will be worked on in an exercise.   Get me up and running quickly.

So those are the types of things that will peak my interest on a submission.  It’s not so much the underlying technology (i.e. xtext, xml, emf, css, javascript, json, java, scala, osgi, etc), but what does it enable and improve upon.  Tell me how it improves or challanges the status quo.

I look forward to your many submissions.

Posted in eclipse, eclipsecon | 2 Comments

Turmeric SOA Distributions

One of the challenges with any open source project, is making it easy to get up and running.  A user should be able to download a binary, and type one command to get started.   Until recently with the Turmeric SOA project, it wasn’t this simple.   Just take a look at the instructions for getting started with the Turmeric Runtime.  If it isn’t easy for somebody to get up and running in about 15 minutes or less, then you have probably already lost a potential adopter.

To help address this, the Turmeric SOA project has started to create some pre-configured open source application servers with the basic turmeric configuration already installed.  These servers are ready for testing and development purposes, and provide a guide how particular services can be deployed.   They will share a common Turmeric Runtime configuration, and will allow for a much smaller WAR file to be created for the services.

The server that we support out of the box is Jetty.  Specifically we are currently providing a pre-configured Jetty 7.4.5 application server configuration for Turmeric SOA, with a sample echoservice deployed.   Currently there is no binary to be downloaded, but during the next release cycle of Turmeric SOA we will be providing this binary distribution for download as well.

The source repository is available on GitHub, with simple instructions on how to get started.  The initial version uses the released Turmeric SOA 1.0.0 runtime.

To build the distribution and sample war yourself, follow these basic steps.

  • Clone the repository from github
  • Make sure you have the settings.xml deployed to your .m2 directory or saved as a file somewhere.
  • Install Maven 3.0 if not already installed.
  • Change to the directory you cloned maven, and type: mvn clean install

If you saved the settings.xml file to your hard drive, you can use the alternative form:

mvn clean install -s path/to/settings.xml

That is it.  The binary distribution for the jetty-turmeric appserver will be located in the app-server/jetty-turmeric/target directory.  Just unzip and follow the rest of the instructions in the README file for the project repository.

Going forward, we are open to community contributions to support deploying and running Turmeric SOA for other application servers.  If you want something else supported, please feel free to fork the project and provide patches back.

Update: Development Snapshots of the jetty distribution can be downloaded from here.

Posted in eclipse, open source, turmeric, web services, wsdl, xsd | 4 Comments