DO Proxy Protocol broken header

[dottodot]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:
0.24.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Digital Ocean

What happened:
Digital Ocean now allows for use of Proxy Protocol
https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/#proxy-protocol
So I've added the annotation to my service

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

and updated my config as follows

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "true"
  enable-brotli: "true"
  enable-vts-status: "true"

However once I've applied these changes I get lots of errors such as the following

6���Zك7�g̮["\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.35.0, server: 0.0.0.0:443
2019/04/11 13:02:57 [error] 265#265: *4443 broken header: "����p�����ўL��k+
rbO-
/�Ç���y�8\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.41.0, server: 0.0.0.0:443
2019/04/11 13:02:57 [error] 265#265: 4444 broken header: "���5�Kk��4 ��b�pxLJw�]��G�V��� �
\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
�98" while reading PROXY protocol, client: 10.244.41.0, server: 0.0.0.0:443

Digital Ocean's response to these error was

This type of response is typically caused by Nginx not properly accepting the PROXY protocol. You should be able to simply append proxy_protocol to your listen directive in your server definition. More information on this can be seen in Nginx's documentation available here:

https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/

I can't see what I'm missing

What you expected to happen:
No errors

[aledbf]

However once I've applied these changes I get lots of errors such as the following
6���Zك7�g̮["\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�

It seems HTTPS traffic is being sent to the HTTP port, please check the port mapping in the ingres-nginx service and your DO console

[dottodot]

@aledbf my service ports are

  ports:
  - name: http
    nodePort: 32342
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31346
    port: 443
    protocol: TCP
    targetPort: https

and the port forwarding on my load balancer is set to
TCP on port 80 > TCP on port 32342
TCP on port 443 > TCP on port 31346

[Routhinator]

Can confirm this issue, same configuration here across the board.

[tlaverdure]

Looking for a solution to this as well. All public requests work but internal traffic to a host of the ingress fail.

2019/05/23 19:59:51 [error] 411#411: *870261 broken header: "��*EpS�M;I��K��WT}�:^ͼ�0���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.2.1, server: 0.0.0.0:443

[dperetti]

It works for me using the helm chart with the following values:

## nginx configuration
## https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml

## https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol
## https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer
controller:
  config:
    use-proxy-protocol: "true"

  service:
    externalTrafficPolicy: "Local"
    annotations:
      # https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/
      # https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

[tlaverdure]

I've got a similar setup. Which is working if I access the host publicly. However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)

Ingress

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
  tls:
  - hosts:
    - 'example.com'
    - '*.example.com'
    secretName: example-tls
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-app
          servicePort: 80
  - host: api.example.com
    http:
      paths:
      - backend:
          serviceName: example-backend
          servicePort: 80
...

Config

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  compute-full-forwarded-for: "true"
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

Curl Example Error

$ curl -v https://api.example.com
* Rebuilt URL to: https://api.example.com/
*   Trying (123.456.789.000...
* TCP_NODELAY set
* Connected to api.example.com (123.456.789.000) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to api.example.com:443 
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to api.example.com:443 

Log from failed request

[error] 661#661: *981009 broken header: "�/�9��ނ���R�6ަ�@%Qe�lG�3.���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.3.1, server: 0.0.0.0:443

[dperetti]

I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" should be set in the Ingress annotation. It has probably no effect here.
My understanding is it must be set on the nginx-ingress service. I think it simply tells DO to activate the "Use Proxy Protocol" setting when the load balancer is created.

[aledbf]

However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)

@tlaverdure that's because you are no specifying the flag --haproxy-protocol in the curl command. If you enable proxy protocol in the ingress controller you need to decode it.

[aledbf]

I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" should be set in the Ingress annotation. It has probably no effect here.

That is correct.

[dottodot]

@dperetti I have service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" on the nginx-ingress service and get the same issue as @tlaverdure

@aledbf I'm not using curl command but get the same errors.

[dperetti]

I also have: externalTrafficPolicy: Local on the nginx-ingress-controller service and of course use-proxy-protocol: "true" in the configmap.
My Kubernetes version is 1.14.1-do.2 .

[tlaverdure]

@dperetti thanks for the tip. I think I added it initially when testing and forgot to remove after I added that annotation to the ingress-nginx service.

@aledbf I'm experiencing this with any type of server side http request. Curl was used to verify the issue, but any server side scripting language that makes a http request to the host (i.e. node) is failing.

[dottodot]

I also have externalTrafficPolicy: Local so don't think that's related.

[dperetti]

Stupid check, but if you go to the load balancer's settings in the DO admin panel, it's enabled, right?
Because if it's not while use-proxy-protocol = true you end up with the same kind of encoding mismatch.