Resolve connectivity issues in an application hosted on an AKS cluster - Azure (2023)

  • Article

Connectivity issues on a Microsoft Azure Kubernetes Service (AKS) cluster can mean different things. In some cases this could mean that the API server connection is affected (for example when using kubectl). In other cases, it could mean that common connectivity issues are affecting an application hosted on the AKS cluster. This article describes how to troubleshoot AKS cluster connectivity issues.

Observation

To troubleshoot common issues when trying to connect to the AKS API server, seeBasic troubleshooting of cluster connectivity issues with the API server.

prerequisites

  • Client URL (curling) or a similar command-line tool.

  • Oapt-getcommand line tool to manipulate packages.

  • Kuberneteskubectltool or a similar tool to connect to the cluster. To install kubectl usingCLI do Azure, run itaz aks install-climandate.

factors to consider

This section covers troubleshooting steps to take if you experience problems trying to connect to your application hosted on an AKS cluster.

In any network scenario, administrators should consider the following important factors when troubleshooting:

  • What is the source and destination of a request?

    (Video) Troubleshoot AKS cluster issues with AKS Diagnostics and AKS Periscope | Azure Friday

  • What is the hop between origin and destination?

  • What is the request-response flow?

  • Which heels have extra layers of security on top, like the following:

    • Firewall
    • Network Security Group (NSG)
    • network policy

When checking each item,receive and parse HTTP response codes. These codes are helpful in determining the nature of the problem and are especially useful in scenarios where the application is responding to HTTP requests.

If other troubleshooting steps are inconclusive, obtain packet captures from both the client and the server. Packet captures are also useful when non-HTTP traffic is involved between the client and the server. See the following Data Collection Guide articles for more information on how to collect packet captures for an AKS environment:

  • Log a TCP dump from a Linux node to an AKS cluster.

  • Register a TCP dump from a Windows node to an AKS cluster.

  • Capture TCP Packets from a Pod in an AKS Cluster.

Knowing how to get HTTP response codes and receive packets makes troubleshooting network connectivity issues easier.

Basic network flow for applications on AKS

In general, the flow of requests to access applications hosted on an AKS cluster is as follows:

Client >> DNS Name >> AKS Load Balancing IP Address >> AKS Nodes >> Pods

There are other possible situations where additional components may be involved. For example:

(Video) How to troubleshoot common issues with AKS Diagnostics

  • The application portal is used throughoutApp sport inputs controller(AGIC) instead of Azure Load Balancer.
  • Azure Front Door and API Management can be used on the load balancer.
  • The process uses an internal load balancer.
  • The connection might not terminate at the requested pod and URL. This might depend on whether the pod can connect to another device, such as a database or another service on the same cluster.

It's important to understand the request flow for the application.

A basic request flow for applications on an AKS cluster will look like the flow shown in the diagram below.

Troubleshooting from the Inside Out

Troubleshooting connection issues can involve a lot of checking, buton the wrong sideThe approach can help find the source of the problem and identify the bottleneck. In this approach, you start in the pod itself and verify that the application responds to the pod's IP address. Then check each item in turn all the way to the final customer.

Step 1: Verify the pod is working and the application or container inside the pod is responding correctly

To determine if the pod is running, do one of the followingkubectl i getcommander:

# List pods in the specified namespace.kubectl get pods -n# List pods em todos os namespaces.kubectl get pods -A

What if the pod doesn't work? In that case, check the pod events usingkubectl describemandate:

kubectl description pod-n

If the patch is not in aPrepareoracestatus or restart several times, checkkubectl describeProduction. Events will reveal any issues that prevent you from starting the pod. Or, if the pod starts, the application inside the pod may have failed, causing the pod to restart.Correct the pod issues accordinglyto ensure it is in good condition.

If the pod is working, it can also be helpful to check the pod's logs and the containers within them. Run the following sequence oflogs kubectlcommander:

logs kubectl-n# Check single container logs in pod.kubectl multicontainer logs-n-c# Dump pod logs (stdout) for a previous container instance.kubectl log--prev # Despeja containerlog (stdout, multicontainerbox) para logs anteriores do container instance.kubectl-c--Earlier

Is the pod working? In that case, test the connection by starting a test group on the cluster. From the test pool, you can directly access the app pod's IP address and verify that the app is responding correctly. run itrun kubectl,apt-get, ecurlingcommands as follows:

# Start a test cluster in the cluster:kubectl run -it --rm aks-ssh --image=debian:stable# After running the test cluster, you will have access to the pod.# You can then run the following commands : apt- get update -y && apt-get install dnsutils -y && apt-get install curl -y# After installing the packages, test the connection to the application pod:curl -Iv http://:

For applications that listen on other protocols, you can install the appropriate tools in the test pool and then test the connection to the application pool.

For more commands for debugging pods, seePod debug mode.

Step 2: Check if the app is available for the service

For scenarios where the application is running inside the pod, you can focus primarily on troubleshooting how the pod is exposed.

(Video) Azure Kubernetes Service : Troubleshooting Part-1

Is the pod exposed as a service? In that case, check the service events. Also make sure the pod's IP address and application port are available as an endpoint in the service description:

# Check the service details.kubectl get svc -n# Describe the service.kubectl describe svc-n

Verify that the pod's IP address exists as an endpoint in the service, as shown in the following example:

$ kubectl get pods -o wide # Verifique o endereço IP do pod.NAME READY STATUS RESTART AGE IP NODE my-pod 1/1 Running 0 12m 10.244.0.15 aks-agentpool-000000-vmss000000 $ service-ip-declu service Check service endpoints . Name: my-cluster-ip-serviceNamespace: defaultSelector: app=my-podType: ClusterIPIP Family Policy: SingleStackIP Families: IPv4IP: 10.0.174.133IPs: 10.0.174> <0.1 /TCPTargetPort: 80 / TCPEndpoints:84 # <-- - Here$ kubectl get endpoints # Verifique os endpoints diretamente para verificação.NAME ENDPOINTS AGEmy-cluster-ip-service 10.244.0.15:80 14m

If the endpoints don't show the pod's correct IP address, check itHang tagseYou choosefor pod and service.

Are the service endpoints correct? In this case, go to the service and check if the application is available.

Access the ClusterIP service

ForClusterIPservice, you can start a test group on the cluster and access the service's IP address:

# Start a test cluster in cluster:kubectl run -it --rm aks-ssh --image=debian:stable # After the test cluster runs, you should be able to access the pod.# Then you can run the following commands: apt -get update -y && apt-get install dnsutils -y && apt-get install curl -y # After installing the packages, test the connection to the service: curl -Iv http://:

If the previous command does not return an appropriate response, check the service events for errors.

Access to the LoadBalancer service

Forload balancerservice, you can access the IP address of the load balancer outside the cluster.

curl -Iv http://:

made it upload balancerDoes the service IP address return the correct answer? If it doesn't, follow these steps:

  1. Confirm service events.

  2. Make sure that the network security groups (NSGs) associated with the AKS nodes and the AKS subnet allow inbound traffic on the service port.

    (Video) Deploy a web application to Azure Kubernetes Service (AKS) cluster

For more commands to debug services, seedebugging services.

Scripts that use input instead of service

For scenarios where the application is exposed using aProhibitedfeature, the traffic flow looks like the following development:

Client >> DNS Name >> Load Balancer or Application Gateway IP Address >> Cluster Inbound Groups >> Service Groups or Group

You can also use the inside-out approach to troubleshooting here. You can also check input and input controller details for more information:

$ kubectl get -n# Check admission and event details CATEGORY FINANCIAL NAME ADDRESSES PORTS AGEhello-world-ingressmyapp.com 20.84.x.x 80, 443 7d22h$ kubectl beskriver inghello-world-ingressName: hello-world-ingressNamespace:Address: 20.84.x.xBackend default: default-http-backend:80 ()TLS: tls-secret terminates myapp.comRules: Host Path Backend ---- ---- -------- myapp.com /blog blog-service:80 (10.244.0.35:80) /store store -service:80 (10.244.0.33:80) Comments: cert-manager.io/cluster-issuer: letsencrypt kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/rewrite target: /$1 nginx .ingress. kubernetes.io/use-regex: trueEvents: Type Cause Age From Notification ---- ------ ---- ---- ------- Regular Sync 5m41s nginx-ingress- controller Scheduled for sync Normal sync 5m41s nginx-ingress-controller Scheduled for sync

This example contains aProhibitedfeature that:

  • he hearsminapp.comsummer.
  • he has twoStistrings have been configured.
  • routes for twoservicesin the rear.

Make sure the backend services are running and responding on the port specified in the input descriptor:

$ kubectl get svc -nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP-PORT(S) main-entry blog service ClusterIP 10.0.155.15480/TCP ingress-basic butik-service Cluster .61.18580/TCP ingress-basic nginx-ingress-ingress-nginx-controller LoadBalancer 10.0.122.148 20.84.x.x 80:30217/TCP,443:32464/TCP

Check the ingress controller pods logs for an error:

$ kubectl get pods -n# Hent ingress controller pods.NAME READY STATUS REBOOT AGEaks-helloworld-one-56c7b8d79d-6zktl 1/1 Running 0 31haks-helloworld-two-558bbv1 Running 0 31hnginx-ingress-ingress-nginx-d5cqn-9d/1 Running 0 31hnginx- ingresso-entrada-nginx-controller-9d8d5c57d-grzdr 1/1 Cor 0 Controle $ 31$b. -n entrada-basic nginx-ingress-ingress-nginx-controller-9d8d5c57d-9vn8q

What if the client requests the input hostname or IP address, but no entries appear in the input controller podlogs? In this case, requests cannot reach the cluster and the user may receive aThe connection has expirederror message.

Another possibility is that components on top of the input pools, such as Load Balancer or Application Gateway, are not correctly routing requests to the cluster. If that's the case, you can check the backend configuration for these features.

If you receive oneThe connection has expirederror message, check the network security group associated with the AKS nodes. Also check the AKS subnet. It can block traffic from the load balancer or application gateway to the AKS nodes.

For more information on how to debug input (such as Nginx Ingress), seeingress-nginx troubleshooting.

(Video) Access to AKS control plane (public, private, vnet integration)

If you have any questions or need help,create a support request, or askAzure community support. You can also submit product feedback atAzure community support.

Disclaimer for contact with third parties

Microsoft provides third-party contact information to help you find additional information about this topic. These contact details are subject to change without notice. Microsoft does not guarantee the accuracy of third-party contact information.

FAQs

How do I troubleshoot network issues in AKS? ›

Ensure that your client's IP address is within the ranges authorized by the cluster's API server:
  1. Find your local IP address. For information on how to find it on Windows and Linux, see How to find my IP.
  2. Update the range that's authorized by the API server by using the az aks update command in Azure CLI.
Oct 24, 2022

How to check connectivity from AKS cluster? ›

Follow these steps:
  1. Connect to Azure Kubernetes Service (AKS) cluster nodes for maintenance or troubleshooting.
  2. Test the DNS resolution to the endpoint: Console Copy. ...
  3. Check the resolv.conf file to determine whether the expected name servers are added: Bash Copy.
Feb 6, 2023

How we will connect to our AKS cluster? ›

Connect to the cluster
  1. Install kubectl locally using the az aks install-cli command. Azure CLI Copy. ...
  2. Configure kubectl to connect to your Kubernetes cluster using the az aks get-credentials command. This command executes the following operations: ...
  3. Verify the connection to your cluster using the kubectl get command.
May 4, 2023

How can you ensure that traffic between the Azure Kubernetes service AKS control plane and the node pools remains in your private network virtual network )? ›

By using a private cluster, you can ensure network traffic between your API server and your node pools remains on the private network only. The control plane or API server is in an Azure Kubernetes Service (AKS)-managed Azure resource group. Your cluster or node pool is in your resource group.

How do you resolve network connectivity problems? ›

  1. Unplug the power cable for the router from the power source.
  2. Unplug the power cable for the modem from the power source. Some modems have a backup battery. ...
  3. Wait at least 30 seconds or so. ...
  4. Plug the modem back into the power source. ...
  5. Plug your router back into the power source. ...
  6. On your PC, try to connect again.

How do I troubleshoot connectivity issues in Azure? ›

Troubleshooting steps
  1. Step 1: Check whether NIC is misconfigured. ...
  2. Step 2: Check whether network traffic is blocked by NSG or UDR. ...
  3. Step 3: Check whether network traffic is blocked by VM firewall. ...
  4. Step 4: Check whether VM app or service is listening on the port. ...
  5. Step 5: Check whether the problem is caused by SNAT.
Feb 10, 2023

How do I check cluster connectivity? ›

To find the cluster IP address of a Kubernetes pod, use the kubectl get pod command on your local machine, with the option -o wide . This option will list more information, including the node the pod resides on, and the pod's cluster IP. The IP column will contain the internal cluster IP address for each pod.

How do you check node connectivity? ›

To check connectivity of a graph, we will try to traverse all nodes using any traversal algorithm. After completing the traversal, if there is any node, which is not visited, then the graph is not connected.

How to check connectivity between two pods in Kubernetes? ›

We will use a Deployment to ensure that iperf3 pods are running on each node in the cluster.
  1. Prerequisites. A running Kubernetes cluster. ...
  2. Step 1: Deploy Iperf3 as a Deployment. ...
  3. Step 2: Verify the Deployment. ...
  4. Step 3: Test Connectivity Between Pods. ...
  5. Conclusion.
Jan 24, 2023

Which command connects to an AKS cluster? ›

To connect to the AKS nodes, you use kubectl debug or the private IP address. This article shows you how to create a connection to an AKS node and update the SSH key on an existing AKS cluster.

How do I create a service connection for AKS? ›

Staging Service Connection
  1. Go to Project -> azure-devops-github-acr-aks-app1 -> Project Settings -> Pipelines -> Service Connections.
  2. Click on New Service Connection.
  3. Choose a service or connection type: kubernetes.
  4. Authentication Method: Azure Subscription.
  5. Username: Azure Cloud Administrator.

How do you communicate between two Kubernetes clusters? ›

How do Pods communicate in Kubernetes?
  1. In Kubernetes, each Pod has an IP address. A Pod can communicate with another Pod by directly addressing its IP address, but the recommended way is to use Services. ...
  2. In Kubernetes, each Pod has its own IP address. ...
  3. Multiple containers in the same Pod share the same IP address.
May 17, 2022

How will you handle the network traffic in Azure? ›

In this module, you will:
  1. Identify the routing capabilities of an Azure virtual network.
  2. Configure routing within a virtual network.
  3. Deploy a basic network virtual appliance.
  4. Configure routing to send traffic through a network virtual appliance.

How do I ensure Azure AKS cluster monitoring is enabled? ›

To enable monitoring of your AKS cluster in the Azure portal from Azure Monitor: In the Azure portal, select Monitor. Select Containers from the list. On the Monitor - containers page, select Unmonitored clusters.

How do you ensure network connectivity? ›

How can I secure my internet connection?
  1. Use strong passwords.
  2. Keep everything updated.
  3. Rename routers and networks.
  4. Turn on encryption.
  5. Use a VPN.
  6. Use multiple firewalls.
  7. Turn off the WPS setting.

Which two tools can be used to troubleshoot Azure Files connectivity? ›

  • Python.
  • .NET.
  • JavaScript.
  • Java.
  • Go.
Apr 13, 2023

How to troubleshoot a connectivity issue between two servers? ›

Fortunately, you can resolve this issue in a few troubleshooting steps.
  1. Step 1: Check for Conflicts in Your DHCP Server. Your network may have more than one DHCP server. ...
  2. Step 2: Check for Duplicate IP Addresses. ...
  3. Step 3: Identify Conflicting MAC Addresses. ...
  4. Step 4: Locate the Switch Ports of Your Conflicting Devices.
Jan 3, 2022

Which command is used to troubleshoot network connectivity? ›

NSLOOKUP

The NSLOOKUP command is used to troubleshoot network connectivity issues in the system. Using the nslookup command, we can access the information related to our system's DNS server, i.e., domain name and IP address.

What is cluster connectivity? ›

Connectivity-based clustering, as the name shows, is based on connectivity between the elements. You create clusters by building a hierarchical tree-type structure. This type of clustering is more informative than the unstructured set of flat clusters created by centroid-based clustering, such as K-means.

How do I find the IP address of a Kubernetes cluster? ›

How to get the IP address of a Kubernetes pod
  1. Go to the kubectl command-line tool. ‍ ...
  2. Run the “kubectl get pods” command. Once you have installed the kubectl command-line tool, run the below command on Kubernetes node to find the pod's name. ...
  3. Copy the pod's name and run the “kubectl get pods server deployment” command.
Jul 18, 2022

How do you check connectivity? ›

Select the Start button, then type settings. Select Settings > Network & internet. The status of your network connection will appear at the top.

How do I check network issues in node? ›

Test network connectivity between nodes
  1. Log in to the Linux shell using SSH.
  2. Ping each of the other nodes in the cluster.
  3. Ping another machine that exists outside of the cluster, for example, a machine that you will use to stage data to be loaded.

How can I test a connection between two IP addresses? ›

Article Details
  1. Open Windows PowerShell through the Start menu.
  2. Enter the command test-netconnection IPAddress -port XXXXX. ...
  3. Press Enter.
  4. Wait for the test to complete.
  5. If the result is True then there is nothing blocking communication between the client and server.
Aug 21, 2020

How do you troubleshoot a failed network? ›

I always start troubleshooting using these simple network troubleshooting steps to help diagnose and refine the issue.
  1. Check the hardware. ...
  2. Use ipconfig. ...
  3. Use ping and tracert. ...
  4. Perform a DNS check. ...
  5. Contact the ISP. ...
  6. Check on virus and malware protection. ...
  7. Review database logs.
Sep 23, 2019

What is network troubleshooting command? ›

You can run common network troubleshooting commands such as arp, ping, ping6, traceroute, traceroute6, NSlookup, and AvgRTTs from the admin console. You can use these connectivity tools to see the network path from the system to a specified server.

How do I troubleshoot Azure App Service Connection? ›

You can also use the Network troubleshooter to troubleshoot the connection issues for the apps in the App Service. To open the network troubleshooter, go to the app service in the Azure portal. Select Diagnostic and solve problem, and then search for Network troubleshooter.

How do I troubleshoot services in Kubernetes? ›

If so, check for the following:
  1. Image name—ensure the image name in the pod manifest is correct.
  2. Image available—ensure the image is really available in the repository.
  3. Test manually—run a docker pull command on the local machine, ensuring you have the appropriate permissions, to see if you can retrieve the image.

What should you do to identify the cause of the connectivity issues? ›

Fortunately, you can resolve this issue in a few troubleshooting steps.
  1. Step 1: Check for Conflicts in Your DHCP Server. Your network may have more than one DHCP server. ...
  2. Step 2: Check for Duplicate IP Addresses. ...
  3. Step 3: Identify Conflicting MAC Addresses. ...
  4. Step 4: Locate the Switch Ports of Your Conflicting Devices.
Jan 3, 2022

What are the 7 troubleshooting steps? ›

Troubleshooting methodologies vary, but the following seven steps are often used.
  1. Gather information. ...
  2. Describe the problem. ...
  3. Determine the most probable cause. ...
  4. Create a plan of action and test a solution. ...
  5. Implement the solution. ...
  6. Analyze the results. ...
  7. Document the process.

What command is used to test network connectivity? ›

ping (This command will test for the Internet connectivity and DNS functionality.) Example: ping www.netgear.com, ping google.com.

What are the six steps in the troubleshooting process? ›

  1. Step One: Define the Problem. Step One is about diagnosing the problem – the context, background and symptoms of the issue. ...
  2. Step Two: Determine the Root Cause(s) of.
  3. Step Three: Develop Alternative Solutions. ...
  4. Step Four: Select a Solution. ...
  5. Step Five: Implement the Solution. ...
  6. Step Six: Evaluate the Outcome.

What are the three main troubleshooting tools? ›

Types of network troubleshooting tools

Ping. Tracert/ Trace Route. Ipconfig/ ifconfig.

How do I test my Azure service connection? ›

Sign in to your organization ( https://dev.azure.com/{yourorganization} ) and select your project.
  1. Select Project settings > Service connections.
  2. Select the service connection that you want to edit.
  3. See the Overview tab of the service connection where you can see the details of the service connection.
Nov 28, 2022

How do I verify my Azure service connection? ›

Go to Project settings > Service connections, and then select the service connection you want to modify. Select Edit in the upper-right corner, and the select Verify. Select Save.

How do I Diagnose and solve problems in Azure Webapp? ›

Open App Service diagnostics

In the left navigation, click on Diagnose and solve problems. For Azure Functions, navigate to your function app, and in the top navigation, click on Platform features, and select Diagnose and solve problems from the Resource management section.

Videos

1. AKS and Azure Load Balancer: How to Expose Your App to the Internet
(CloudSkills)
2. How to create AKS Cluster 2023 | Azure Kubernetes Service | Azure Container Service | K21 Academy
(Docker & Kubernetes with K21Academy)
3. Monitoring Azure Kubernetes Service (AKS) with Azure Monitor
(Shailender Choudhary)
4. How to create AKS Cluster | Setting Up & Deploying an Azure Kubernetes Service Cluster | K21Academy
(K21Academy)
5. AKS Kubenet networking explained in plain English - in less than 5 minutes
(cloud-monk - cloud in plain english)
6. Troubleshooting instance related issues on Azure App Services
(Microsoft Azure)
Top Articles
Latest Posts
Article information

Author: Barbera Armstrong

Last Updated: 06/03/2023

Views: 6055

Rating: 4.9 / 5 (79 voted)

Reviews: 94% of readers found this page helpful

Author information

Name: Barbera Armstrong

Birthday: 1992-09-12

Address: Suite 993 99852 Daugherty Causeway, Ritchiehaven, VT 49630

Phone: +5026838435397

Job: National Engineer

Hobby: Listening to music, Board games, Photography, Ice skating, LARPing, Kite flying, Rugby

Introduction: My name is Barbera Armstrong, I am a lovely, delightful, cooperative, funny, enchanting, vivacious, tender person who loves writing and wants to share my knowledge and understanding with you.