JDunphy wrote:This is quite the mystery.
Is there a local ip table here? That sendto failing to 127.0.0.1 is really suspicious. The bgread tells us that it was an udp packet but that the socket had a read error when it attempted to recv from it.
If you have iptables on this box, have you verified that you don't have a bunch of nf_conntrack: table full, dropping packet" in syslog.
What is odd is that I get that the client resolver might not get any answer from your local resolver. I do not get why your local resolver would refuse the connection.
It feels like a FW issue. Have you any dynamic rules like ipsets that could be firing from unsual acitivty loads. I know one of the tricks we sometime see is with spammers with tons of NS records and then rejecting our resolvers attempts to verify the reverse so we go down the line looking for the next server. It puts on quite the 'RCODE refused' show with named. LOL
Code: Select all
Sep 21 05:54:30 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 211.139.178.48#53
Sep 21 05:54:30 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 211.136.192.12#53
Sep 21 05:54:30 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 120.196.165.40#53
Sep 21 05:54:35 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 211.139.178.48#53
Sep 21 05:54:35 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 211.136.192.12#53
Sep 21 05:54:35 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 120.196.165.40#53
Sep 21 05:54:37 relay10 named[971]: error (unexpected RCODE REFUSED) resolving '18.186.239.120.in-addr.arpa/PTR/IN': 211.139.178.48#53
Our noc has a swatch monitor up watching for syslog patterns and it will turn red when these errors are displayed because we track errors with red on the monitor. I saw one last week with 20+ NS records and they connected repeatedly before our dynamic ipset rules put them in timeout for 4 hr. Always something isn't it?
Thanks everybody for your susgests!
I was upset for 3 days from this issue raises, SA includes DCC, DNS Blocklist and Razor, Pyzor seem cannot connect to get data, so the system immediately was attacked by SPAM, even the SPF module also doesnot work when it always said that: Gmail has no correct SPF record......
Many many errors related Host not found or Domain cannnot lookup although i change DNS master anymore.....
With "Operation not permitted" i think it's related permission but zmfixperms don't think like this, everything is good.
With "dns: bad dns reply" this clue drives me to check the queries to DNS but when i manually queried, it's seem OK.
Finally, i found that some queries are good, some queries are bad because the firewall block UDP due to a Outbound UDP Flood rule. This rule limit 20 connections/s.
I disabled this rule for 24h till now, everything come back like before. SPAM was denied by DNS Blocklist and DCC, Razor + Pyzor score count again and SPF module works as a gate keeper.
The problem was RESOLVED, thanks everybody so much!