Part 2 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?


Elasticsearch Data node setup :

Data node are the nodes/instance where our index data will reside.There is two things to
consider for data nodes while creating an index i.e
1.Number of shards:
How to divide our index data into number of shards so that data can be retrieved in fast
paced manner. It help in sharding data to different nodes so that we can scale it well.
2.Number of Replicas:
Number of Copies of primary shards are called number of replicas

Let’s take an example to understand it’s concept :

How many node failure can be possible if we have 3 node data cluster with 5 shards and 1 replica.
Elasticsearch notes:
Primary shards and its replica will not be on same node.





If we delete one node from our cluster then :
Our cluster will re-balance our lost replicas and move to existing nodes ,for time-being the cluster becomes yellow as you can see below for unassigned shards.


Once rebalancing is completed our cluster back to green state.


Answer for our Question:
Our cluster can sustain two node failure if we have 3 data node with index
(5 shards and 1 replica) and hence our cluster will be in yellow state.

Comments

Post a Comment

Popular posts from this blog

Part 4 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?

Image
Shard allocation concept can help us to use spot instance on Elasticsearch Data nodes: By using rack feature in elasticsearch node we can tell that primary shard to be on one rack and its replica shard to be on another rack. Like as per our elasticsearch setup we have divided ondemand rack and spot rack as per below elasticsearch.yml configuration. elasticsearch.yml for ondemand rack: cluster.name: my-application-1 node.name: ondemand path.data: /data/elasticsearch-1-2 path.logs: /var/log/elasticsearch http.port: 9200 node.master: false node.data: true node.attr.rack_id: ondemand cluster.routing.allocation.awareness.attributes: rack_id discovery.zen.ping.unicast.hosts: ["nb-master-es1.testing.com", "nb-master-es2.testing.com",”nb-master-es3.testing.com”] elasticsearch.yml for onspot rack: cluster.name: my-application-1 node.name: onspot path.data: /data/elasticsearch-1-5 path.logs: /var/log/elasticsearch http.port
Read more

What is a Devops ??

What is Devops and its culture to adopt? Devops is simply a means of collaboration between developers and operations team working together over a common goal of making a commit change or complete source code live with just click of an instance i.e How operations team with the help of their coding skill sets automate their infrastructure in an cost efficient and effective manner in order to save time to market the product in fastest possible way. Devops Best Practices:(AWS Specific)    1.Do not stick with any single tool or code for solving a problem whether it can be Backup and restore or monitoring,cloud infrastructure setup ,first solve the problem then provide the best way of doing it with a purpose like for monitoring we can use any tool like nagios or zabbix but both tools has its own advantages and disadvantages so before moving with any tools it better to understand its pros and cons. 2.Always Keep your App server,DB server in private subnet so no public access. 3.Mo
Read more

Part 1 Series of Elasticsearch Cluster on Spot Instances with zero downtime in App?

Image
Elasticsearch is an open source ,distributable,scalable and enterprise grade search engine. It can powers up your web-app search mechanism with n-number of queries executing and fetching results for your end user within milliseconds. Use case of Elasticsearch: 1.Full Text search 2.Log Analysis in ELK stack 3.Scraping and Combining Public Data 4.Event Data and Metrics(Time series is good option) Will Start with Discussion for Elasticsearch Master node Setup : We will be creating a three master node cluster in order to keep quorum so that there will not be any split brain problem. Split brain problem usually occurs in case of two master nodes cluster setup and one node goes down then second node will treat itself to be a candidate for master node form a cluster and suddenly first node comes up and itself treat as a master and form a new es cluster.So two nodes for two cluster which will not be rejoin until one is restarted again. The above pr
Read more