DevOPS Learn and Share

Shard allocation concept can help us to use spot instance on Elasticsearch Data nodes: By using rack feature in elasticsearch node we can tell that primary shard to be on one rack and its replica shard to be on another rack. Like as per our elasticsearch setup we have divided ondemand rack and spot rack as per below elasticsearch.yml configuration. elasticsearch.yml for ondemand rack: cluster.name: my-application-1 node.name: ondemand path.data: /data/elasticsearch-1-2 path.logs: /var/log/elasticsearch http.port: 9200 node.master: false node.data: true node.attr.rack_id: ondemand cluster.routing.allocation.awareness.attributes: rack_id discovery.zen.ping.unicast.hosts: ["nb-master-es1.testing.com", "nb-master-es2.testing.com",”nb-master-es3.testing.com”] elasticsearch.yml for onspot rack: cluster.name: my-application-1 node.name: onspot path.data: /data/elasticsearch-1-5 path.logs: /var/log/elasticsearch http.port...

Elasticsearch Note for cluster State: Red state: There is no primary shard in the cluster for indexing ,writing and searching Yellow State: There is primary shard for indexing and searching but not replica shards available of primary shards. Green state: Primary shard and replica shard are in good state for indexing and searching. Question:What happen when we increase the number of replicas: curl -XPUT 'localhost:9200/testing/_settings?pretty' -H 'Content-Type: application/json' -d' { "index" : {    "number_of_replicas" : 2 } } '               As you can see above the replica 2’s for primary shards will be unassigned as we have 2 data node cluster so let’s increase the data node in our setup. So from our conclusion we can see that by increasing the number of replicas in our case we can handle 2 node failur...

Elasticsearch Data node setup : Data node are the nodes/instance where our index data will reside.There is two things to consider for data nodes while creating an index i.e 1.Number of shards: How to divide our index data into number of shards so that data can be retrieved in fast paced manner. It help in sharding data to different nodes so that we can scale it well. 2.Number of Replicas: Number of Copies of primary shards are called number of replicas Let’s take an example to understand it’s concept : How many node failure can be possible if we have 3 node data cluster with 5 shards and 1 replica. Elasticsearch notes: Primary shards and its replica will not be on same node. If we delete one node from our cluster then : Our cluster will re-balance our lost replicas and move to existing nodes ,for time-being the cluster becomes yellow as you can see below for unassigned shards. Once rebalancing is completed our cluster back to green state...

Elasticsearch is an open source ,distributable,scalable and enterprise grade search engine. It can powers up your web-app search mechanism with n-number of queries executing and fetching results for your end user within milliseconds. Use case of Elasticsearch: 1.Full Text search 2.Log Analysis in ELK stack 3.Scraping and Combining Public Data 4.Event Data and Metrics(Time series is good option) Will Start with Discussion for Elasticsearch Master node Setup : We will be creating a three master node cluster in order to keep quorum so that there will not be any split brain problem. Split brain problem usually occurs in case of two master nodes cluster setup and one node goes down then second node will treat itself to be a candidate for master node form a cluster and suddenly first node comes up and itself treat as a master and form a new es cluster.So two nodes for two cluster which will not be rejoin until one is restarted again. The above pr...

What is Devops and its culture to adopt? Devops is simply a means of collaboration between developers and operations team working together over a common goal of making a commit change or complete source code live with just click of an instance i.e How operations team with the help of their coding skill sets automate their infrastructure in an cost efficient and effective manner in order to save time to market the product in fastest possible way. Devops Best Practices:(AWS Specific)    1.Do not stick with any single tool or code for solving a problem whether it can be Backup and restore or monitoring,cloud infrastructure setup ,first solve the problem then provide the best way of doing it with a purpose like for monitoring we can use any tool like nagios or zabbix but both tools has its own advantages and disadvantages so before moving with any tools it better to understand its pros and cons. 2.Always Keep your App server,DB server in private subnet so no public acces...