ImageThis tutorial by zewaren.net shows us how to get ElasticSearch mappings shared with files in FreeBSD.

Original post: http://zewaren.net/site/?q=node/148

Using the API

You could specify the index mappings using the PUT mapping API, but you’d have to do that every time:

Create the index with the mapping:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
    "mappings" : {
        "awesome_doctype" : {
            "properties" : {
              "a_mysql_date" : {
                "type" : "date",
                "format" : "yyyy-MM-dd HH:mm:ss"
              },
              "a_string" : {
                "type" : "string",
                "analyzer" : "french"
              },
              "a_long" : {
                "type" : "long"
              },
              "a_boolean" : {
                "type" : "boolean"
              }
            }
        }
    }
}'

Put something into the index:

1
2
3
4
5
6
7
{
    "a_mysql_date": "2015-02-02 22:22:22",
    "a_string": "Je suis une saucisse",
    "a_long": 4242,
    "a_boolean": 1
}'

Querying the mapping confirms that it’s correct:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "awesome_index" : {
    "mappings" : {
      "awesome_doctype" : {
        "properties" : {
          "a_boolean" : {
            "type" : "boolean"
          },
          "a_long" : {
            "type" : "long"
          },
          "a_mysql_date" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss"
          },
          "a_string" : {
            "type" : "string",
            "analyzer" : "french"
          }
        }
      }
    }
  }
}

Storing the mappings in files

The quest of the home folder

The documentation and people online say that JSON mapping files can be put in folder:

1
$ES_HOME/config/mappings/_default

But where on earth is that folder???

Maybe it’s user elasticsearch‘s home folder on the system.

1
2
# getent passwd elasticsearch
elasticsearch:*:965:965:elasticsearch user:/nonexistent:/usr/sbin/nologin

Nope, the user doesn’t have a home.

On mailing lists, it appears that ES_HOME is where elasticsearch was built. Since I installed it using the ports, that does not make sense to me.

Maybe querying the nodes of the server will give me the info:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# curl 'localhost:9200/_nodes?pretty'
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "v0rj_yxBTW2XnbChNL__EA" : {
      "name" : "Father Time",
      "transport_address" : "inet[/203.0.113.42:9300]",
      "host" : "dbs3.example.com",
      "ip" : "203.0.113.42",
      "version" : "1.4.2",
      "build" : "927caff",
      "http_address" : "inet[/203.0.113.42:9200]",
      "settings" : {
        "pidfile" : "/var/run/elasticsearch.pid",
        "path" : {
          "work" : "/var/tmp/elasticsearch",
          "data" : "/var/db/elasticsearch",
          "conf" : "/usr/local/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch",
          "plugins" : "/usr/local/lib/elasticsearch/plugins"
        },
        "cluster" : {
          "name" : "elasticsearch"
        },
        "config" : "/usr/local/etc/elasticsearch/elasticsearch.yml",
        "client" : {
          "type" : "node"
        },
        "name" : "Father Time"
      },
[...]
}

The instance does not have a home configured. Hum.

Let’s add one in elasticsearch.yml:

1
path.home: /usr/home/elasticsearch

Restarting the server and querying the node again confirms that the parameter was understood.

Having the mapping file loaded

Let’s add a mapping file in our configured path:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# cat /usr/home/elasticsearch/config/mappings/_default/awesome_doctype.json
{
    "awesome_doctype" : {
        "properties" : {
          "a_mysql_date" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss"
          },
          "a_string" : {
            "type" : "string",
            "analyzer" : "french"
          },
          "a_long" : {
            "type" : "long"
          },
          "a_boolean" : {
            "type" : "boolean"
          }
        }
    }
}'

Let’s delete the index, add a doc and see if the correct mapping was loaded automatically:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{"acknowledged":true}
    "a_mysql_date": "2015-02-02 22:22:22",
    "a_string": "Je suis une saucisse",
    "a_long": 4242,
    "a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}
{
  "awesome_index" : {
    "mappings" : {
      "awesome_doctype" : {
        "properties" : {
          "a_boolean" : {
            "type" : "long"
          },
          "a_long" : {
            "type" : "long"
          },
          "a_mysql_date" : {
            "type" : "string"
          },
          "a_string" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

It wasn’t!

The only remaining solution is to trace the instance and see where it REALLY looks for that mysterious mapping folder.

1
2
3
4
5
6
7
8
9
10
11
# ktrace -d /usr/local/openjdk7/bin/java -Des.pidfile=/var/run/elasticsearch.pid -server -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOcc
upancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch -Des.config=/usr/local/etc/elasticsearch/elasticsearch.yml -cp /usr/local/li
b/elasticsearch/elasticsearch-1.4.2.jar:/usr/local/lib/elasticsearch/*:/usr/local/lib/elasticsearch/sigar/* org.elasticsearch.bootstrap.Elasticsearch
[...]
# kdump | grep mappings
[...]
  3700 java     CALL  stat(0x8602998e0,0x7ffffe181668)
  3700 java     NAMI  "/usr/local/etc/elasticsearch/mappings"
  3700 java     RET   stat -1 errno 2 No such file or directory
[...]

Haha! It’s in etc!

Let’s move the mapping there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# cat /usr/local/etc/elasticsearch/mappings/_default/awesome_doctype.json
{
    "awesome_doctype" : {
        "properties" : {
          "a_mysql_date" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss"
          },
          "a_string" : {
            "type" : "string",
            "analyzer" : "french"
          },
          "a_long" : {
            "type" : "long"
          },
          "a_boolean" : {
            "type" : "boolean"
          }
        }
    }
}

Let’s try the experiment again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{"acknowledged":true}
    "a_mysql_date": "2015-02-02 22:22:22",
    "a_string": "Je suis une saucisse",
    "a_long": 4242,
    "a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}
{
  "awesome_index" : {
    "mappings" : {
      "awesome_doctype" : {
        "properties" : {
          "a_boolean" : {
            "type" : "boolean"
          },
          "a_long" : {
            "type" : "long"
          },
          "a_mysql_date" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss"
          },
          "a_string" : {
            "type" : "string",
            "analyzer" : "french"
          }
        }
      }
    }
  }
}

Success!

What needs to be remembered here is that the mappings aren’t in the home folder. They’re in the conf folder. The fact that the conf folder is in the home folder is just mere coincidence on many systems.