Part 4 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?




Shard allocation concept can help us to use spot instance on Elasticsearch Data nodes:
By using rack feature in elasticsearch node we can tell that primary
shard to be on one rack and its replica shard to be on another rack.
Like as per our elasticsearch setup we have divided ondemand rack
and spot rack as per below elasticsearch.yml configuration.


elasticsearch.yml for ondemand rack:
cluster.name: my-application-1
node.name: ondemand
path.data: /data/elasticsearch-1-2
path.logs: /var/log/elasticsearch
http.port: 9200
node.master: false
node.data: true
node.attr.rack_id: ondemand
cluster.routing.allocation.awareness.attributes: rack_id
discovery.zen.ping.unicast.hosts: ["nb-master-es1.testing.com", "nb-master-es2.testing.com",”nb-master-es3.testing.com”]
elasticsearch.yml for onspot rack:
cluster.name: my-application-1
node.name: onspot
path.data: /data/elasticsearch-1-5
path.logs: /var/log/elasticsearch
http.port: 9200
node.master: false
node.data: true
node.attr.rack_id: spot
cluster.routing.allocation.awareness.attributes: rack_id

discovery.zen.ping.unicast.hosts: ["nb-master-es1.testing.com", "nb-master-es2.testing.com",”nb-master-es3.testing.com”]


We can see shards well dispersed by keeping replica as 1 so we
can see below that are data is divided into primary and replica
shards evenly distributed between ondemand and onspot
rack configured nodes.


Lets stop all our spot instances and then check the cluster health:

So from above snapshot we can see if our spot instances goes
away our cluster will be in yellow state and our searching and
indexing functionality will not be impacted as our primary
shard is healthy.Hence we can run our Elasticsearch cluster
on spot instances in order to save $ xxxx from our current
ondemand Infrastructure.


Reference:
https://www.objectrocket.com/blog/elasticsearch/top-elasticsearch-use-cases/

Comments

Post a Comment

Popular posts from this blog

What is a Devops ??

What is Devops and its culture to adopt? Devops is simply a means of collaboration between developers and operations team working together over a common goal of making a commit change or complete source code live with just click of an instance i.e How operations team with the help of their coding skill sets automate their infrastructure in an cost efficient and effective manner in order to save time to market the product in fastest possible way. Devops Best Practices:(AWS Specific)    1.Do not stick with any single tool or code for solving a problem whether it can be Backup and restore or monitoring,cloud infrastructure setup ,first solve the problem then provide the best way of doing it with a purpose like for monitoring we can use any tool like nagios or zabbix but both tools has its own advantages and disadvantages so before moving with any tools it better to understand its pros and cons. 2.Always Keep your App server,DB server in private subnet so no public acces...
Read more

Part 1 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?

Image
Elasticsearch is an open source ,distributable,scalable and enterprise grade search engine. It can powers up your web-app search mechanism with n-number of queries executing and fetching results for your end user within milliseconds. Use case of Elasticsearch: 1.Full Text search 2.Log Analysis in ELK stack 3.Scraping and Combining Public Data 4.Event Data and Metrics(Time series is good option) Will Start with Discussion for Elasticsearch Master node Setup : We will be creating a three master node cluster in order to keep quorum so that there will not be any split brain problem. Split brain problem usually occurs in case of two master nodes cluster setup and one node goes down then second node will treat itself to be a candidate for master node form a cluster and suddenly first node comes up and itself treat as a master and form a new es cluster.So two nodes for two cluster which will not be rejoin until one is restarted again. The above pr...
Read more

Part 3 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?

Image
Elasticsearch Note for cluster State: Red state: There is no primary shard in the cluster for indexing ,writing and searching Yellow State: There is primary shard for indexing and searching but not replica shards available of primary shards. Green state: Primary shard and replica shard are in good state for indexing and searching. Question:What happen when we increase the number of replicas: curl -XPUT 'localhost:9200/testing/_settings?pretty' -H 'Content-Type: application/json' -d' { "index" : {    "number_of_replicas" : 2 } } '               As you can see above the replica 2’s for primary shards will be unassigned as we have 2 data node cluster so let’s increase the data node in our setup. So from our conclusion we can see that by increasing the number of replicas in our case we can handle 2 node failur...
Read more