Your site is blocked from a region and you don´t know why? Maybe SNI related?

Your site is blocked from a region and you don´t know why? Maybe SNI related?

The other day at work, we received issues from customers. They told us that they couldn´t access to their web Instance. In all the cases, the origin was the same country.

The first thing that I ask, is to give me a screenshot of the error, but it wasn´t a great help.

We also asked for a telnet instance 443

My surprise was that, telnet worked well, but then….

With curl doesn´t work well. Maybe It´s pointing to something related to SSL.

We decided to sniff on the client side, loadbalancer, etc…

We could see RSTs just after the client Client Hello. It indicates that it could be a problem with the handshake, maybe ciphers, etc etc..

Here I don´t have the loadbalancer .pcap files, but as far as I remember, in the loadbalancers we also received RSTs. So what´s sending the RSTs?

Let´s examine the Client Hello inside the .pcap file.

Let´s go down until Server Name Indication extension. After server Name length. It should be the name of the severname (I deleted in this case)

What´s the Server Name Indication Extension: Server Name Indication, often abbreviated SNI, is an extension to TLS that allows multiple hostnames to be served over HTTPS from the same IP address.

Let´s try to change the headers with curl or openssl.

Here you could see how it worked. Why? Let´s go and open the new .pcap file.

In this new .pcap we can´t see the the SNI extension and we can´t also see any of the RST

Ok, if we have more than one site in that ip address it would be a mess, because we would access to the first loaded certificate, but in this case we only wanted to see that the problem was with SNI filtering.

Here you have a nice website to check a website from different agents. In this case agents in China and other regions, to see the differences.

https://www.websitepulse.com/tools/china-firewall-test

SNI filtering is used very often by Internet providers to block access to torrent sites or similar. In this case it might be related to the great firewall (China).

Networking Problem: I can’t connect to your service (tcp) failed: Connection timed out

Networking Problem: I can’t connect to your service (tcp) failed: Connection timed out

Imagine that a friend is trying to connect to one of your services and he mention that when he tries to connect, finally displays a  “(tcp) failed: Connection timed out”

The first thing, I go and check if I could connect to the service, then I’ll check if the service is working properly, if it’s right, I will go and check the firewall…..

Wow, I have all open in iptables, everybody could connect to that service, but I need to deal with my friend and tell him something! Because he told me, that he doesn’t have any rule that could block the connections.

First of all, I’m going to try to simulate this problem.

I open the port listening in X ip.

nc -l 127.0.0.2 3000

 

Then I start sniffing:

tcpdump -vvv -s0 -i lo -w lo.pcap

 

With netcat I also try to connect to the service:

nc -v -z 127.0.0.2 3000
nc: connect to 127.0.0.2 port 3000 (tcp) failed: Connection timed out

 

And now I open the .pcap with wireshark.

127.0.0.1 is my FRIENDS IP and 127.0.0.2 is the service in port 3000.

Here we could see, how my FRIEND/CLIENT send me a SYN, but when I answer with the SYN,ACK the client send me a retransmission of the SYN, and here it’s where the loop starts, because I also have to send him again a SYN,ACK.

The first thing that I think: the origin is blocking the incoming SYN,ACK

 

So I ask my friend for the RULES, and here they are:

 

Changing the limits on the fly

Changing the limits on the fly

I never did this before, but now that I know that it works, I will do it more often.

To show how it works, I changed the number of processes (nproc) hard limit of the user koteo in the limits.conf to 10

With the prlimit command I display the limits of the first PID of user “koteo”, that matches with the limits of the file limits.conf

Now I execute bash until I get the error “Cannot fork”.

I execute prlimit with the parameter –nproc=1024:1024 (soft:hard) and the parameter –pid $pid (we get $pid from pgrep) . We just changed the soft and hard limit to 1024, as you could see at the bottom of the next screenshot.

Now I can execute again the bash command, after the error “Cannot fork”.

Here I show you, that we actually have more than 10 processes for koteo user.

 

If I’m not wrong, this started working with kernels 2.6.32+

I know that in few versions of the kernel works “echo -n “Max processes=SOFT_L:HARD_L” > /proc/$PID/limits” but not in the one that I have. It displays the error:

`write(2, “: Invalid argument”, 18: Invalid argument) = 18`

 

 

List of tools added to project Menu

List of tools added to project Menu

Here you could see some of them:

Tools

Query your webservices with a simple python script

Query your webservices with a simple python script

Hello all!!

I usually need yo check webservices. I check the http code and the time it takes to give me back the result of the soap query. It works only with ssl.

To execute the code:

python ./query_webservice.py -file /tmp/file.xml -host ws.example.com -context /context/ws -soapv v1.1

You will need to have a valid file.xml and specify the soap version

The output result:

HTTP_CODE: 200 HEALTH: OK
Exec_Total_Time: 35 ms

#!/usr/bin/env python
#-*- coding: utf-8 -*-
# Koldo Oteo - (koteo [at] sitereliabilityengineer.io)
# December 18th 2017
import sys, time
import argparse
import httplib
import xml.dom.minidom

### Parse arguments
parser = argparse.ArgumentParser(description='Example:  ./query_webservice.py -file /tmp/file.xml \
                                 -host hostname.domain -context /context -soapv v1.1')
parser.add_argument('-file', action='store', dest='xml',
                    help='xml File Name')
parser.add_argument('-host', action='store', dest='host',
                    help='Webservice host')
parser.add_argument('-context', action='store', dest='context',
                    help='Webservice context')
parser.add_argument('-soapv', action='store', dest='soapv',
                    help='Soap Version v1.1 or v1.2')
# Print Parser Help
if len(sys.argv) == 1:
    parser.print_help()
    sys.exit(1)
param = parser.parse_args()
###

### FUNCTION TO Read xml File
def read_xml():
   with open(param.xml, 'r') as f:
      xmlmsg = f.read()
      return xmlmsg

###

### FUNCTION TO POST XML TO WEBSERVICE
def post_xml(xmlmsg):
   """HTTP XML Post request"""
   if param.soapv == "v1.2":
      headers = {"Content-type": "application/soap+xml","Content-Length": "%d" % len(xmlmsg), "charset": "utf-8", "SOAPAction": "", "User-Agent": "PythonSOAPClient"}
   elif param.soapv == "v1.1":
      headers = {"Content-type": "text/xml","Content-Length": "%d" % len(xmlmsg), "charset": "utf-8", "SOAPAction": "", "User-Agent": "PythonSOAPClient"}
   conn = httplib.HTTPSConnection(param.host)
   conn.request("POST", param.context, "", headers)
   # Send xml
   conn.send(xmlmsg)
   response = conn.getresponse()
   print "HTTP_CODE: %s  HEALTH: %s" % (response.status, response.reason)
   data = response.read()
   #resultxml = xml.dom.minidom.parseString(data)
   #print (resultxml.toprettyxml())
   conn.close()

###
# READ XML FILE
xmlmsg = read_xml()

# GET EXECUTION TOTAL TIME AND POST XML
start_time = time.time()
post_xml(xmlmsg)
print("Exec_Total_Time: %s ms" % int(round((time.time() - start_time) * 1000)))


###