zmconfigd: Can't create listener socket: [Errno -1] Unmapped exception: SocketException: Protocol family unavailable

Zimbra Collaboration 8.8 Beta
Locked
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

zmconfigd: Can't create listener socket: [Errno -1] Unmapped exception: SocketException: Protocol family unavailable

Post by msquadrat »

I've seen this issue now two times while doing test installs with the current beta; it doesn't happen every time though:

The setup takes ages at the "Starting servers" part and suddenly an emergency message from journald pops up:

Code: Select all

Setting zimbraSmtpHostname for ip-172-31-4-202.eu-central-1.compute.internal...done.
Configuring SNMP...done.
Setting up syslog.conf...done.
Enabling IMAP protocol for zimbra-imapd service...done.
Enabling IMAPS protocol for zimbra-imapd service...done.
Disabling zimbraNetworkModulesNGEnabled 
Disabling zimbraNetworkMobileNGEnabled 
Starting servers...
Broadcast message from systemd-journald@ip-172-31-4-202 (Thu 2017-08-31 14:43:44 UTC):

zmconfigd[22171]: Can't create listener socket: [Errno -1] Unmapped exception: java.net.SocketException: Protocol family unavailable
/var/log/zimbra.log says:

Code: Select all

Aug 31 14:43:38 ip-172-31-4-202 zimbramon[22101]: 22101:info: Starting zmconfigd via zmcontrol
Aug 31 14:43:43 ip-172-31-4-202 zmconfigd[22171]: zmconfigd started on ip-172-31-4-202.eu-central-1.compute.internal with loglevel=3 pid=22171
Aug 31 14:43:43 ip-172-31-4-202 zmconfigd[22171]: Fetching All configs
Aug 31 14:43:43 ip-172-31-4-202 zmconfigd[22171]: Skipping Configuration for server  update.
Aug 31 14:43:43 ip-172-31-4-202 zmconfigd[22171]: gs <function getserver at 0x5>(None) 1 java.lang.NullPointerException
Aug 31 14:43:43 ip-172-31-4-202 zmconfigd[22171]: All configs fetched in 0.38 seconds
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: All rewrite threads completed in 0.01 sec
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: All restarts completed in 0.00 sec
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: Error creating listener socket on port 7171: [Errno -1] Unmapped exception: java.net.SocketException: Protocol family unavailable
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: Service unavailable ([Errno 20001] getaddrinfo failed)
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: Can't create listener socket: [Errno -1] Unmapped exception: java.net.SocketException: Protocol family unavailable
Aug 31 14:43:44 ip-172-31-4-202 zmconfigd[22171]: zmconfigd: shutting down
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmconfigd: Can't create listener socket: [Errno -1] Unmapped exception: SocketException: Protocol family unavailable

Post by msquadrat »

Looks like something is really wrong with zmconfigd, it randomly crashes with hung threads on rewrite requests:

Code: Select all

Aug 31 15:11:48 ip-172-31-4-202 zmconfigd[22355]: Processing rewrite request for proxy
Aug 31 15:11:53 ip-172-31-4-202 zmconfigd[22355]: join() Thread-480 FAILED
Aug 31 15:11:53 ip-172-31-4-202 zmconfigd[22355]: Hung threads detected (3 total), exiting

Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: zmconfigd started on ip-172-31-4-202.eu-central-1.compute.internal with loglevel=3 pid=29107
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Fetching All configs
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Skipping Configuration for server  update.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: gs <function getserver at 0x5>(None) 1 java.lang.NullPointerException
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Skipping All Reverse Proxy URLs update.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: getAllReverseProxyURLs <function garpu at 0x3>(None) 1 __getattribute__ not found on type LdapServer. See http://bugs.jython.org/issue2487 for details.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Skipping All Reverse Proxy Backends update.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: getAllReverseProxyBackends <function garpb at 0x6>(None) 1 __getattribute__ not found on type LdapServer. See http://bugs.jython.org/issue2487 for details.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Skipping All Memcached Servers update.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: getAllMemcachedServers <function gamcs at 0x7>(None) 1 __getattribute__ not found on type LdapServer. See http://bugs.jython.org/issue2487 for details.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Skipping All MTA Authentication Target URLs update.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: getAllMtaAuthURLs <function gamau at 0x8>(None) 1 __getattribute__ not found on type LdapServer. See http://bugs.jython.org/issue2487 for details.
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: All configs fetched in 0.38 seconds
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: All rewrite threads completed in 0.01 sec
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: All restarts completed in 0.00 sec
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Error creating listener socket on port 7171: [Errno -1] Unmapped exception: java.net.SocketException: Protocol family unavailable
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Service unavailable ([Errno 20001] getaddrinfo failed)
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: Can't create listener socket: [Errno -1] Unmapped exception: java.net.SocketException: Protocol family unavailable
Aug 31 15:12:40 ip-172-31-4-202 zmconfigd[29107]: zmconfigd: shutting down
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmconfigd: Can't create listener socket: [Errno -1] Unmapped exception: SocketException: Protocol family unavailable

Post by msquadrat »

The Jython bug is titled "PyType.fromClass publication bug on slow systems". This might be caused by AWS, maybe I ran out of CPU credits.

But I can still reliably kill zmconfigd by doing a zmproxyctl restart which always results in:

Code: Select all

Aug 31 15:26:53 ip-172-31-4-202 zmconfigd[5672]: Processing rewrite request for proxy
Aug 31 15:26:58 ip-172-31-4-202 zmconfigd[5672]: join() Thread-324 FAILED
Aug 31 15:26:58 ip-172-31-4-202 zmconfigd[5672]: Hung threads detected (3 total), exiting
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmconfigd: Can't create listener socket: [Errno -1] Unmapped exception: SocketException: Protocol family unavailable

Post by msquadrat »

It looks like there are a few issues in zmconfigd. I opened [bug]108305[/bug] on the join() issue and I also ran into [bug]108294[/bug] which DualBoot already reported and where zmconfigd doesn't answer on the second and any further STATUS requests because the socket is half-closed somehow:

Code: Select all

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 37000)
Traceback (most recent call last):
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/SocketServer$py.class", line 601, in process_request_thread
    self.finish_request(request, client_address)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/SocketServer$py.class", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/SocketServer$py.class", line 657, in __init__
    self.handle()
  File "/opt/zimbra/common/lib/jylibs/listener.py", line 61, in handle
    self.request.send(response)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/_socket$py.class", line 1484, in send
    return super(ChildSocket, self).send(data)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/_socket$py.class", line 382, in handle_exception
    return method_or_function(*args, **kwargs)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/_socket$py.class", line 382, in handle_exception
    return method_or_function(*args, **kwargs)
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/_socket$py.class", line 1200, in send
    self._handle_channel_future(future, "send")
  File "/opt/zimbra/lib/jars/jython-standalone-2.7.1.jar/Lib/_socket$py.class", line 384, in handle_exception
    raise _map_exception(jlx)
error: [Errno 104] Socket closed
----------------------------------------
Locked