Rancher Service Discovery Hidden Gems

Alena Prokharchyk
Alena Prokharchyk
Gray Calendar Icon Published: March 9, 2016
Gray Calendar Icon Updated: September 11, 2019

Since the Service Discovery feature was first introduced in May 2015, Rancher engineering has never stopped adding new functionality and improvements to it. Rancher Betausers and forumsparticipants have been sharing their applications and architecture details to help us shape the product to cover more use cases. This article reviews some of those very useful features.

Rolling Restart

Through the lifecycle of your Service Discovery, there can be a need to restart it after reapplying certain configuration changes. Usually, the number one requirement is to have no service outage while the restart is happening. The rolling restart feature makes this magic happen and is now available in Rancher Compose (UI support is coming soon). There are a couple of optional parameters must be set for the restart action to initiate: Rolling
restart1 Once these options are set, the service’s containers with these settings will be restarted in a batch size of \“1\” followed by a batch restart at a “zero ms interval\” since the last batch restart completion.

Random public port assignment to service

When the service is deployed to a particular port, it’s important that public ports defined in the service don’t conflict with the ports that are already in use in this host. Maintaining and tracking this information can be an unnecessary inconvenience. Random public port assignment becomes useful in this situation. Let’s say you want to deploy a nginx service with scale=2 that internally listens to port 80. You know that you do have at least three hosts in the system, but are not aware of which ports are already allocated. All you need to do is offload the public port picking decision to Rancher by omitting to specify a public port when to adding a service. Rancher will choose the available public port from 49153-65535 range (configurable) and this port will be published on all three hosts where the service is deployed.

randomPortServiceCreate Rancher choses the available public port from 49153-65535 range (configurable) and this port will be published on ALL 3 hosts where the service is deployed: randomPortServiceCreated


Load Balancer service custom config

The Rancher Load Balancer service allows redistribution of traffic between target services. The services can join and leave the load balancer at any time user demands it. Traffic can then be forwarded to them based on the user’s defined host name routing rules. However, we’ve always wanted to give the user even more flexibility in configuring the Load Balancer, as specific apps might require specific Load Balancer configuration tweaks. The Load Balancer service custom configuration feature will enable it. Let’s say we want to:

  • Switch LB algorithm from default \“Round Robin\” to \“Least connection.” So, the server with the lowest number of connections receives the connection
  • Set \“httpclose\” mode so LB works in HTTP tunnel mode and checks to see if a \“Connection: close\” header is already set in each direction, and will add one if it is missing
  • Tune up the performance by increasing the number of max per-process concurrent connections

To do that, on the \“Add Load Balancer\” dialog, go to the \“Custom haproxy.cfg\” tab and set the following three parameters customHaproxyConfig1 And set the following 3 parameters: custom haproxy cfg
2 Upon Load Balancer service startup, the custom values will be set in Load Balancer config file. You can find more about what custom properties can be configured at this url: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html

\“Start once\” service option

By default, the service reconcile code requires all service containers to be in “running” state. If any of the containers dies right after the start, or some time later, it will be recreated on the spot. For most applications, that is the default-required behavior for most, but not all of them. For example, short-lived containers that are meant to start, finish the task and exit without coming back later. Alternatively, data containers that just have to be created, but staying in a running state are not a requirement for them. Rancher enables this sort of behavior with \“Start once\” option. To enable it, on the \“Add service\” dialog expand \“Advanced Options\” and on the very first tab pick \“Start once.\” startOnce All containers that were created and stopped, will never go to the running state again unless the user explicitly choses to start them via Container->Start.

Health check strategies

No doubt, the health check configuration is defined by application parameters and requirements (port, health check endpoint, number of retries till application is considered alive). But what about recovering from a failed health check? Should it just be as simple as recreating a failed container? Not necessarily, it also can be driven by your application needs. Rancher Health Check strategies enable this functionality. To try this feature, go to Applications view, click on \“Add Service\” and expand \“Advanced options\” on the dialog opened. Navigate to \“Health Check \” tab, and pick either \“TCP...\” or \“HTTP...\” setting. Screen Shot 2016-03-07 at 8.54.31
PM The \“When unhealthy\” status provides you with three choices:

  1. Re-create, which is the default option - when a container goes unhealthy, it will be re-created by Rancher service reconcile code.
  2. Take no action - sometimes you just want the health check to act like a monitoring tool by simply displaying a \“healthy\“/\“unhealthy\” state for a container without taking any further action. Perhaps you want to take a look at the failure cause, and recover the container manually. Or suppose it’s a testing environment. The failed health check is really proof that your application reacted correctly to some failure-causing-trigger in your application.
  3. Re-create only when least (n) container is healthy - this is useful for applications like Zookeeper where maintaining a certain quorum (replicated group of container) is essential. Dropping a number of running containers below quorum can cause application failures. Lets say your service has six containers running, with the quorum of three. Configuring the health check strategy with \“Recreate only when at least three containers are healthy\” would ensure that number of running containers would never drop below this number.

Today, three strategies are enabled, but we are planning to expand in the future. Please share your opinions on what strategy your application could use, on Rancher forums.

Contact us

If you’re interested in finding out more about Rancher, please download the software from github, join the Betato get additional support, and visit our forums to view topics and ask questions. To learn more about using Rancher, please join our next online meetup. Avatar Alena Prokharchyk If you have any questions or feedback, please contact me on twitter: @lemonjet https://github.com/alena1108

Alena Prokharchyk
Alena Prokharchyk
Senior Manager, Engineering
Alena is a Principal Software Engineer at Rancher Labs, who’s been working on building infrastructure services first for Virtual Machines, now for containers with main focus on Kubernetes. She enjoys helping others make sense of problems and explore solutions together. In her free time Alena enjoys rollerblading, reading books on totally random subjects and listening to other people’s stories.
Get started with Rancher