Missing logs in Elasticsearch logs at midnight

Case: of the Missing logs

I was debugging a curious case of my Elasticsearch instance on my vagrant dev box going to RED state every night at 00:00:00.  Consistently as far back as I can remember.

Right the obvious thing to do is look at the logs right? Except for this set of rotated logs there are no lines between 23:40hrs to 00:00:05.  Not in the current un-rotated log or the previous set.

At First Pass:

  1. Elasticsearch rotates its own log.  Could it be this process causing the missing Elasticsearch log lines?
  2. Marvel Creates new daily indices at 00:00:00.  Could it be this causing the missing Elasticsearch log lines?

What was the real was causing the missing logs

Well By default Elasticsearch uses log4j.  However, instead of the standard log4j.property file you get with log4j Elasticsearch is using a translated format to YAML format excluding all of the log4j pre-fix giveaways.  Another closer look at the configuration lead to the curious investigation of the type of rolling appender ; DailyRollingFile. This lead to this revelation :

DailyRollingFileAppender extends FileAppender so that the underlying file is rolled over at a user chosen frequency. DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss. The log4j extras companion includes alternatives which should be considered for new deployments and which are discussed in the documentation for org.apache.log4j.rolling.RollingFileAppender.

Source :  Apache’s DailyRollingFileAppender Documentation

Missing Elastic logs Root Cause:

The sync issue with the DailyRollingFileAppender must be the cause to the missing Elasticsearch log lines around midnight.

Missing Elastic logs fix:

Use a log4j alternatives to DailyRollingFileAppender.  In this case a RollingFileAppender, changing my rolling strategy to roll my logs when they reach a certain file size. Replace DailyRollingFileAppender with RollingFileAppender and removing the  datePattern which was for the DailyRollingFileAppender.

Example:

file:
    type: rollingFile
    file: ${path.logs}/${cluster.name}.log
    maxFileSize: 10000000
    maxBackupIndex: 10
    layout:
        type: pattern
        conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Note: YAML is particular about tabs!

Happy Ending

Marvel turns out to be the cause of the my Elasticsearch cluster going into RED state at mid-night on new .Marvel*** Index creation.  Which makes sense as there will be a few milliseconds-seconds when this new index will have been created with shards, replicas etc missing.

Lean Maven Release

The Lean Maven Release (AKA Maven Release on Steroids)
Simply put to get rid of Maven Release Plugin’s repetitive and time wasting inefficient builds and multiple checkins to SCM script this process to:
mvn clean
mvn versions:set
mvn deploy
mvn scm:tag
This can be setup in both Jenkins and Team-city.  I have been able configured this within a few minutes replacing my teams Maven Release Plugin.  This is a huge time safer and more people really need to do this especially within true Continuous Delivery team or any type of setting with frequent need for builds.

Benefits

 So how big exactly was the improvement of Releases On Steroids over the Release Plugin?

See for yourself

Releases on Steroids Maven Release Plug-in
Clean/Compile/Test cycle 1 3
POM transformations 0 2
Commits 0 2
SCM revisons
1 (binary source tag)
3

See more at Axel Fontaine’s blog where I first came across this piece of treasure after manager tipped me about it.

 

A Typical Fix

Typically The following is all I ever need to add on any project to getting maven on steriods pattern working

Add the Properties

...
<maven.compiler.plugin.version>3.1</maven.compiler.plugin.version>
<maven.release.plugin.version>2.5</maven.release.plugin.version>
<maven.source.plugin.version>2.2.1</maven.source.plugin.version>
<maven.javadoc.plugin.version>2.9.1</maven.javadoc.plugin.version>
<maven.gpg.plugin.version>1.5</maven.gpg.plugin.version>
...

Deployment path settings

... Local deployment
<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Internal repo</name>
        <url>file:///${user.home}/.m2/repository/internal.local</url>
    </repository>
</distributionManagement>
...
... or Remote deployment
<distributionManagement>
    <repository>
      <uniqueVersion>false</uniqueVersion>
      <id>corp1</id>
      <name>Corporate Repository</name>
      <url>scp://repo/maven2</url>
      <layout>default</layout>
    </repository>
</distributionManagement>

Apache maven plugins

<pluinManagement>
<plugins>
...
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>${maven.compiler.plugin.version}</version>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-release-plugin</artifactId>
    <version>${maven.release.plugin.version}</version>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-source-plugin</artifactId>
    <version>${maven.source.plugin.version}</version>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-javadoc-plugin</artifactId>
    <version>${maven.javadoc.plugin.version}</version>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-gpg-plugin</artifactId>
    <version>${maven.gpg.plugin.version}</version>
</plugin>
...
<plugins>
...
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-release-plugin</artifactId>

    <configuration>
        <useReleaseProfile>false</useReleaseProfile>
        <releaseProfiles>release</releaseProfiles>
        <goals>deploy</goals>
    </configuration>
</plugin>
<plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <configuration>
        <outputDirectory>${project.build.directory}/releases/</outputDirectory>
        <descriptors>
            <descriptor>${basedir}/src/main/assemblies/plugin.xml</descriptor>
        </descriptors>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.3</version>
    <configuration>
        <source>${java.version}</source>
        <target>${java.version}</target>
    </configuration>
</plugin>
<plugin>
    <artifactId>maven-clean-plugin</artifactId>
    <version>2.6.1</version>
    <configuration>
        <filesets>
            <fileset>
                <directory>overlays</directory>
                <includes>
                    <include>**/*</include>
                </includes>
                <followSymlinks>false</followSymlinks>
            </fileset>
        </filesets>
    </configuration>

</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-deploy-plugin</artifactId>
    <version>2.8.2</version>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-source-plugin</artifactId>
    <version>2.4</version>
</plugin>
...

The Release Profile

The Maven Release profile will be infered to for the maven deploy option

<profiles>
...
    <profile>
        <id>release</id>
        <properties>
            <activatedProperties>release</activatedProperties>
        </properties>
        <build>
            <pluginManagement>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-compiler-plugin</artifactId>
                        <version>${maven.compiler.plugin.version}</version>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-release-plugin</artifactId>
                        <version>${maven.release.plugin.version}</version>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-source-plugin</artifactId>
                        <version>${maven.source.plugin.version}</version>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-javadoc-plugin</artifactId>
                        <version>${maven.javadoc.plugin.version}</version>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-gpg-plugin</artifactId>
                        <version>${maven.gpg.plugin.version}</version>
                    </plugin>
                </plugins>
            </pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-source-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>attach-sources</id>
                            <goals>
                                <goal>jar</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-javadoc-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>attach-javadocs</id>
                            <goals>
                                <goal>jar</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-gpg-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>sign-artifacts</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>sign</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>
...


Assemly description

And the plugin assemly description xml.  Location: src/main/assemblies/plugin.xml

<?xml version="1.0" encoding="UTF-8"?>
<assembly>
    <id>plugin</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <useTransitiveFiltering>true</useTransitiveFiltering>
            <excludes>
            </excludes>
        </dependencySet>
    </dependencySets>
</assembly>

Note if you intend to sign the package with the GPG plugin you’ll need to further configure this for your environment. I will write a seperate blog for this later. You may skip it with the bellow:

mvn deploy -Prelease -Dgpg.skip=true

Thats It!

Git Alias Configuration Example

Edit your .gitconfig file in your $HOME directory for some serious time saving git shortcuts

[core]
 excludesfile = /Users/moses.mansaray/.gitignore_global
 autocrlf = input

[user]
 name = moses.mansaray
 email = moses.mansaray@domain.com

[push]
 default = simple

[alias]
 co = checkout
 cob = checkout -b
 cod = checkout develop
 ci = commit
 st = status

 save = !git add -A &amp;&amp; git commit -m
 br = branch

 rhhard-1 = reset --hard HEAD~1
 rhhard-o = reset head --hard

 hist = log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short

 type = cat-file -t
 dump = cat-file -p

 llf = log --pretty=format:"%C(yellow)%h%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate --numstat
 lld = log --pretty=format:"%C(yellow)%h\\ %ad%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate --date=short

 amend = commit -a --amend

Elasticsearch Cheat Sheet and Short Examples

Quick short, Elasticsearch cheat API End-Point calls that takes a while to remember.  If I have missed your favourite or want to make a recommendation to add in please do leave comment

States


# Show all indices
GET /_cat/indices?v-
  
# cluster health state
GET /_cluster/health
  
# Show all nodes
GET /_cat/nodes?

# Show largest Index. Leverages the _CAT api
curl 'localhost:9200/_cat/indices?bytes=b' | sort -rnk8 | grep -v marvel,kibana

Index

Indexing


# Bulk Indexing Example
POST /factory/_bulk
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"swift","make":"suzuki", "mark":1, "release_year":"1998-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"swift","make":"suzuki", "mark":2, "release_year":"2003-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"baleno","make":"suzuki", "mark":1, "release_year":"2000-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"focus","make":"ford","mark":1, "release_year":"2001-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"focus","make":"ford","mark":2, "release_year":"2007-01-01"}
	  {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"rs","make":"ford","mark":2, "release_year":"2011-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"rav4","make":"toyota","mark":3, "release_year":"2009-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"mondeo","make":"ford","mark":2, "release_year":"2007-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
	 { "model":"st","make":"ford","mark":1, "release_year":"2007-01-01"}
	 {"index":{"_index":"factory", "_type":"cars"}}
 { "model":"5 series","make":"bmw","mark":3, "release_year":"2009-01-01"}

 

Index Management


PUT /my_index/_settings
 
{
  "index": {
    "number_of_replicas": 4
  }
}
 
# Add a single alias
PUT /lmg_sem_v4/_alias/lmg
 
 
# Move Shard to another node
POST /_cluster/reroute
{
    "commands" : [ {
        "move" :
            {
              "index" : "amg_sem_v12", "shard" : 0,
              "from_node" : "UK-SEARCH-STG-02", "to_node" : "UK-SEARCH-STG-01"
            }
        }
    ]
}


Index Cloning

From ElasticSearch 2.3 you you may now use the built in _reindex API to re-index data


POST /_reindex
{
  "source": {
    "index": "my-index"
  },
  "dest": {
    "index": "my-new-index"
  }
}

 

Cloning with a filter/query


POST /_reindex
{
  "source": {
    "index": "my-index",
    "query": {
      "term": {
        "has-index-cloning-with-filter-on": true
      }
    }
  },
  "dest": {
    "index": "my-new-index"
  }
}

 


# Show cluster-wide Recovery state
GET /_recovery?pretty&amp;amp;amp;amp;amp;amp;amp;amp;human
GET /_recovery?pretty&amp;amp;amp;amp;amp;amp;amp;amp;human&amp;amp;amp;amp;amp;amp;amp;amp;active_only=true
 
# show tabular cluster-wide status summary
GET /_cat/recovery?v
 
# Show me all snapshots
GET /_snapshot/_all
 
# Show settings details of snapshot repo "my_backup"
GET /_snapshot/my_backup
 
# Show all snapshot details of repo "my_backup"
GET /_snapshot/my_backup/_all
 
# Delete snapshot "snapshot_2015_09_07-13_50_48" from repo "prod-0009"
DELETE /_snapshot/prod-0009/snapshot_2015_09_07-13_50_48/
 
# Register Repo + no need to verify permission on path location
PUT /_snapshot/prod-0009?verify=false
{
   "type": "fs",
   "settings": {
      "location": "/vagrant/prod-0009",
      "compress": true,
      "max_snapshot_bytes_per_sec": "200000000",
      "max_restore_bytes_per_sec": " 500mb"
      }
}
 
# Take Snapshot of just "cmg_sem_v6" index
PUT /_snapshot/one-off-repo?wait_for_completion=true
{
  "indices": "cmg_sem_v6",
  "ignore_unavailable": "true",
  "include_global_state": false
}
 
 
# Restore Snapshots of all index + global state
POST /_snapshot/prod-0009/snapshot_2015_09_11-10_23_29/_restore
 
# Restore Snapshots of only "log_river" index
POST /_snapshot/prod-0009/snapshot_2015_09_11-10_23_29/_restore
{
  "indices": "log_river",
  "rename_pattern": "index_pattern",  
  "rename_replacement": "restored_pattern" 
  "ignore_unavailable": "true",
  "include_global_state": false
}
 
# Speed up Recovery Speed
PUT /_cluster/settings
{
   "persistent": {
      "cluster.routing.allocation.node_concurrent_recoveries": "5",
      "indices.recovery.max_bytes_per_sec": "200mb",
      "indices.recovery.concurrent_streams": 5
   }
}
 

Having trouble With .Marvel* index creation?


# You can view the current settings template with :
curl -XGET localhost:9200/_template/marvel
 
# Modify settings with:
PUT /_template/marvel_custom
{
    "order" : 1,
    "template" : ".marvel*",
    "settings" : {
        "number_of_replicas" : 0,
        "number_of_shards" : 5
    }
}
 

 

More here

Move/Route shards to another elasticsearch node


POST /_cluster/reroute
{
    "commands" : [ {
        "move" :
            {
              "index" : "amg_sem_v12", "shard" : 0,
              "from_node" : "UK-SEARCH-STG-02", "to_node" : "UK-SEARCH-STG-01"
            }
        }
    ]
}