Zimbra Web Service Change

Have a great idea for extending Zimbra? Share ideas, ask questions, contribute, and get feedback.
Post Reply
gromit08
Posts: 9
Joined: Sat Sep 13, 2014 2:53 am
ZCS/ZD Version: 8.7.5_GA_1764.RHEL6_64_201703140325

Zimbra Web Service Change

Post by gromit08 »

During a recent OS update the following zimbra packages were installed:

zimbra-apache-components-1.0.1-1zimbra8.7b1.el6.x86_64
zimbra-core-components-1.0.3-1zimbra8.7b1.el6.x86_64
zimbra-mariadb-10.1.25-1zimbra8.7b1.el6.x86_64
zimbra-mariadb-libs-10.1.25-1zimbra8.7b1.el6.x86_64
zimbra-openjdk-1.8.0u144b01-1zimbra8.7b4.el6.x86_64
zimbra-openssl-1.0.2l-1zimbra8.7b2.el6.x86_64
zimbra-openssl-libs-1.0.2l-1zimbra8.7b2.el6.x86_64
zimbra-php-5.6.31-1zimbra8.7b2.el6.x86_64
zimbra-spell-components-1.0.1-1zimbra8.7b1.el6.x86_64
zimbra-store-components-1.0.1-1zimbra8.7b1.el6.x86_64

Pretty much everything that we were doing with web service calls failed after that. Below is a change that the developers had to make, which appears to be working, initially.
This will involve a lot of code changes to get this to work and appears to be a bug, introduced in one of the updates. Is this a known bug, or is there a fix for this?

Affects GetDistributionListRequest, GetAccountRequest, and others as well.
Previously working:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:zimbra="urn:zimbra" xmlns:zaccount="urn:zimbraAccount" xmlns:zmail="urn:zimbraMail" xmlns:zadmin="urn:zimbraAdmin" >
<soap:Header>
<zimbra:context xmlns="urn:zimbraSoap">
<authToken>0_redacted</authToken>
</zimbra:context>
</soap:Header>
<soap:Body>
<zadmin:GetDistributionListRequest>
<dl by="name">ois@pittstate.edu</dl>
</zadmin:GetDistributionListRequest>
</soap:Body>
</soap:Envelope>


Had to change to get it to work post upgrade:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:zimbra="urn:zimbra" xmlns:zaccount="urn:zimbraAccount" xmlns:zmail="urn:zimbraMail" xmlns:zadmin="urn:zimbraAdmin" >
<soap:Header>
<zimbra:context xmlns="urn:zimbraSoap">
<authToken>0_redacted</authToken>
</zimbra:context>
</soap:Header>
<soap:Body>
<GetDistributionListRequest xmlns="urn:zimbraAdmin">
<dl by="name">ois@pittstate.edu</dl>
</GetDistributionListRequest>
</soap:Body>
</soap:Envelope>
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Zimbra Web Service Change

Post by msquadrat »

You have to make sure that you send actual well-formed XML. Your old code had the issue that you didn't specify the XML namespace for the "dl" element; adding a zadmin: prefix there should have worked as well but I think your approach is cleaner.

Since the SOAP documentation is down right now, maybe the WSDL files are helpful for you: https://wiki.zimbra.com/wiki/Wsdl
shrikant-prasad
Posts: 11
Joined: Fri Jun 09, 2017 4:57 am

Re: Zimbra Web Service Change

Post by shrikant-prasad »

your previously working request is still working fine for me.
What is the response you are getting?
gromit08
Posts: 9
Joined: Sat Sep 13, 2014 2:53 am
ZCS/ZD Version: 8.7.5_GA_1764.RHEL6_64_201703140325

Re: Zimbra Web Service Change

Post by gromit08 »

Thanks for the response.

This is the xml we get back from the first example that fails.

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<context xmlns="urn:zimbra"/>
</soap:Header>
<soap:Body>
<soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>system failure: java.lang.NullPointerException</faultstring>
<detail>
<Error xmlns="urn:zimbra">
<Code>service.FAILURE</Code>
<Trace>qtp1286783232-70093:1510070747810:ef76d270d65bd4ed</Trace>
</Error>
</detail>
</soap:Fault>
</soap:Body>
</soap:Envelope>
shrikant-prasad
Posts: 11
Joined: Fri Jun 09, 2017 4:57 am

Re: Zimbra Web Service Change

Post by shrikant-prasad »

can you please share the stacktrace for this NPE from mailbox.log?
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Zimbra Web Service Change

Post by msquadrat »

It was pretty easy to reproduce with the given request on a Release 8.8.4.GA.1877.UBUNTU16.64 UBUNTU16_64 NETWORK edition setup:

Code: Select all

2017-11-09 08:56:59,649 WARN  [qtp1121172875-15889:https:https://ip-172-31-4-202.eu-central-1.compute.internal:7071/service/admin/soap] [name=zimbra;ip=172.31.4.202;port=39878;] SoapEngine - handler exception
java.lang.NullPointerException
        at com.zimbra.cs.service.admin.GetDistributionList.handle(GetDistributionList.java:94)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:607)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:460)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:273)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:303)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:213)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.server.Server.handle(Server.java:517)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:748)
2017-11-09 08:56:59,650 INFO  [qtp1121172875-15889:https:https://ip-172-31-4-202.eu-central-1.compute.internal:7071/service/admin/soap] [name=zimbra;ip=172.31.4.202;port=39878;] soap - GetDistributionListRequest elapsed=4
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Zimbra Web Service Change

Post by msquadrat »

Something is really odd here. I took a proper request from zmprov --debug and just dropped the dl element. This resulted in a proper SOAP error "invalid request: missing required element: dl". Then I just added a wrong xmlns to the dl element and got the NPE (see below). When I renamed the element but kept the wrong namespace, I got the proper SOAP error again. At least the first and the last one should be equivalent since in both cases a sub-element exists but is in the wrong namespace. The SOAP/XML parser shouldn't look at the elements name at all then.

Code: Select all

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
  <soap:Header>
    <context xmlns="urn:zimbra">
      <authToken>0_xxx</authToken>
      <nosession/>
      <userAgent name="zmprov" version="8.8.4_GA_1877"/>
      <authTokenControl voidOnExpired="1"/>
    </context>
  </soap:Header>
  <soap:Body>
    <GetDistributionListRequest xmlns="urn:zimbraAdmin" xmlns:foo="http://example.com">
        <foo:dl by="name">test@example.com</foo:dl>
    </GetDistributionListRequest>
  </soap:Body>
</soap:Envelope>
shrikant-prasad
Posts: 11
Joined: Fri Jun 09, 2017 4:57 am

Re: Zimbra Web Service Change

Post by shrikant-prasad »

finally found the issue. There hasn't been any changes in Zimbra Soap API. The difference in behaviour (now all elements need to be defined with proper namespace) is because of change in the open jdk version used with zimbra build. now openjdk-1.8.0u144 is used and there was a fix in the jdk code : "8134111: Unmarshaller unmarshalls XML element which doesn't have the expected namespace".
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Zimbra Web Service Change

Post by msquadrat »

Thanks for the info! Did you already open a bug on Bugzilla for the NPE issue this causes so we could subscribe to it? If not, I'd open one.
Post Reply