elastic tutorial

for v0.6.0

elastic is an R client for Elasticsearch. This tutorial is an introduction to the package.

Installation

You can install from CRAN

install.packages("elastic")

Or the development version from GitHub

install.packages("devtools")
devtools::install_github("ropensci/elastic")

Then load the package

library("elastic")

Elasticsearch info

Install Elasticsearch

Unix (linux/osx)

Replace 2.3.2 with the version you are working with.

  • Download zip or tar file from Elasticsearch see here for download, e.g., curl -L -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-2.3.2.tar.gz
  • Uncompress it: tar -xvf elasticsearch-2.3.2.tar.gz
  • Move it: sudo mv elasticsearch-2.3.2 /usr/local
  • Navigate to /usr/local: cd /usr/local
  • Add shortcut: sudo ln -s elasticsearch-2.3.2 elasticsearch

On OSX, you can install via Homebrew: brew install elasticsearch

Windows

Windows users can follow the above, but unzip the zip file instead of uncompressing the tar file.

Start Elasticsearch

  • Navigate to elasticsearch: cd /usr/local/elasticsearch
  • Start elasticsearch: bin/elasticsearch

I create a little bash shortcut called es that does both of the above commands in one step (cd /usr/local/elasticsearch && bin/elasticsearch).

Note: Windows users should run the elasticsearch.bat file

Initialize connection

The function connect() is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by connect() are written to your options for the current session, and are used by elastic functions.

connect()
#> transport:  http
#> host:       127.0.0.1
#> port:      9200
#> headers (names):  NULL
#> username:  NULL
#> password:  NULL
#> errors:    simple
#> Elasticsearch (ES) details:
#>    name:                    Ningal
#>    ES version:              2.3.1
#>    ES version timestamp:    2016-04-04T12:25:05Z
#>    ES build hash:           bd980929010aef404e7cb0843e61d0665269fc39
#>    lucene version:          5.5.0

On package load, your base url and port are set to http://127.0.0.1 and 9200, respectively. You can of course override these settings per session or for all sessions.

Get some data

Elasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It's sort of JSON, but would pass no JSON linter. I include a few data sets in elastic so it's easy to get up and running, and so when you run examples in this package they'll actually run the same way (hopefully).

I have prepared a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads (see below). See elastic:::make_bulk_plos and elastic:::make_bulk_gbif.

Shakespeare data

Elasticsearch provides some data on Shakespeare plays. I've provided a subset of this data in this package. Get the path for the file specific to your machine:

shakespeare <- system.file("examples", "shakespeare_data.json", package = "elastic")

Then load the data into Elasticsearch:

docs_bulk(shakespeare)

If you need some big data to play with, the shakespeare dataset is a good one to start with. You can get the whole thing and pop it into Elasticsearch (beware, may take up to 10 minutes or so.):

curl -XGET http://www.elasticsearch.org/guide/en/kibana/current/snippets/shakespeare.json > shakespeare.json
curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json

Public Library of Science (PLOS) data

A dataset inluded in the elastic package is metadata for PLOS scholarly articles. Get the file path, then load:

plosdat <- system.file("examples", "plos_data.json", package = "elastic")
docs_bulk(plosdat)

Global Biodiversity Information Facility (GBIF) data

A dataset inluded in the elastic package is data for GBIF species occurrence records. Get the file path, then load:

gbifdat <- system.file("examples", "gbif_data.json", package = "elastic")
docs_bulk(gbifdat)

GBIF geo data with a coordinates element to allow geo_shape queries

gbifgeo <- system.file("examples", "gbif_geo.json", package = "elastic")
docs_bulk(gbifgeo)

More data sets

There are more datasets formatted for bulk loading in the ropensci/elastic_data GitHub repository. Find it at https://github.com/ropensci/elastic_data

Search the plos index and only return 1 result

Search(index="plos", size=1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#>
#> [[1]]$`_type`
#> [1] "article"
#>
#> [[1]]$`_id`
#> [1] "0"
#>
#> [[1]]$`_score`
#> [1] 1
#>
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0007737"
#>
#> [[1]]$`_source`$title
#> [1] "Phospholipase C-β4 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"

Search the plos index, and the article document type, sort by title, and query for antibody, limit to 1 result

Search(index="plos", type="article", sort="title", q="antibody", size=1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#>
#> [[1]]$`_type`
#> [1] "article"
#>
#> [[1]]$`_id`
#> [1] "568"
#>
#> [[1]]$`_score`
#> NULL
#>
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0085002"
#>
#> [[1]]$`_source`$title
#> [1] "Evaluation of 131I-Anti-Angiotensin II Type 1 Receptor Monoclonal Antibody as a Reporter for Hepatocellular Carcinoma"
#>
#>
#> [[1]]$sort
#> [[1]]$sort[[1]]
#> [1] "1"

A new function in v0.4 is Search_uri(), where the search is defined entirely in the URL itself. This is especially useful for cases in which POST requests are forbidden, e.g, on a server that prevents POST requests for security reasons (which the function Search() uses)

Basic search

Search_uri(index = "plos", size = 1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#>
#> [[1]]$`_type`
#> [1] "article"
#>
#> [[1]]$`_id`
#> [1] "0"
#>
#> [[1]]$`_score`
#> [1] 1
#>
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0007737"
#>
#> [[1]]$`_source`$title
#> [1] "Phospholipase C-β4 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"

Sorting

res <- Search_uri(index = "shakespeare", type = "act", sort = "speaker:desc", fields = 'speaker')
sapply(res$hits$hits, "[[", c("fields", "speaker"))
#> [[1]]
#> [1] "ARCHBISHOP OF YORK"
#>
#> [[2]]
#> [1] "VERNON"
#>
#> [[3]]
#> [1] "PLANTAGENET"
#>
#> [[4]]
#> [1] "PETO"
#>
#> [[5]]
#> [1] "KING HENRY IV"
#>
#> [[6]]
#> [1] "HOTSPUR"
#>
#> [[7]]
#> [1] "FALSTAFF"
#>
#> [[8]]
#> [1] "CHARLES"
#>
#> [[9]]
#> [1] ""

A bool query

mmatch <- '{
 "query": {
   "bool" : {
     "must_not" : {
       "range" : {
         "speech_number" : {
           "from" : 1, "to": 5
}}}}}}'
sapply(Search(index="shakespeare", body=mmatch)$hits$hits, function(x) x$`_source`$speech_number)
#>  [1]  6  7  7  7  7  8  9 10  7  8

Fuzzy query

Fuzzy query on numerics

fuzzy <- list(query = list(fuzzy = list(speech_number = list(value = 7, fuzziness = 4))))
Search(index="shakespeare", body=fuzzy)$hits$total
#> [1] 1499

Range query

With numeric

body <- list(query=list(range=list(decimalLongitude=list(gte=1, lte=3))))
Search('gbif', body=body)$hits$total
#> [1] 24

With dates

body <- list(query=list(range=list(eventDate=list(gte="2012-01-01", lte="now"))))
Search('gbif', body=body)$hits$total
#> [1] 899

More-like-this query (more_like_this can be shortened to mlt)

body <- '{
 "query": {
   "more_like_this": {
     "fields": ["abstract","title"],
     "like_text": "and then",
     "min_term_freq": 1,
     "max_query_terms": 12
   }
 }
}'
Search('plos', body=body)$hits$total
#> [1] 488

Highlighting

body <- '{
 "query": {
   "query_string": {
     "query" : "cell"
   }
 },
 "highlight": {
   "fields": {
     "title": {"number_of_fragments": 2}
   }
 }
}'
out <- Search('plos', 'article', body=body)
out$hits$total
#> [1] 58
sapply(out$hits$hits, function(x) x$highlight$title[[1]])[8:10]
#> [[1]]
#> NULL
#>
#> [[2]]
#> NULL
#>
#> [[3]]
#> NULL

Scrolling search - instead of paging

Search('shakespeare', q="a*")$hits$total
#> [1] 2747
res <- Search(index = 'shakespeare', q="a*", scroll="1m")
res <- Search(index = 'shakespeare', q="a*", scroll="1m", search_type = "scan")
length(scroll(scroll_id = res$`_scroll_id`)$hits$hits)
#> [1] 50

Bulk load from R objects

A new feature in v0.4 is loading data into Elasticsearch via the bulk API (faster than via the normal route) from R objects (data.frame, or list). E.g.:

Using a pretty large data.frame, at 53K rows, load ggplot2 package first

library("ggplot2")
res <- invisible(docs_bulk(diamonds, "diam"))
Search(index = "diam")$hits$total
#> [1] 158140

Get documents

Get document with id=1

docs_get(index='plos', type='article', id=1)
#> $`_index`
#> [1] "plos"
#>
#> $`_type`
#> [1] "article"
#>
#> $`_id`
#> [1] "1"
#>
#> $`_version`
#> [1] 1
#>
#> $found
#> [1] TRUE
#>
#> $`_source`
#> $`_source`$id
#> [1] "10.1371/journal.pone.0098602"
#>
#> $`_source`$title
#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"

Get certain fields

docs_get(index='plos', type='article', id=1, fields='id')
#> $`_index`
#> [1] "plos"
#>
#> $`_type`
#> [1] "article"
#>
#> $`_id`
#> [1] "1"
#>
#> $`_version`
#> [1] 1
#>
#> $found
#> [1] TRUE
#>
#> $fields
#> $fields$id
#> $fields$id[[1]]
#> [1] "10.1371/journal.pone.0098602"

Get multiple documents at once

Same index and type, different document ids

docs_mget(index="plos", type="article", id=3:4)
#> $docs
#> $docs[[1]]
#> $docs[[1]]$`_index`
#> [1] "plos"
#>
#> $docs[[1]]$`_type`
#> [1] "article"
#>
#> $docs[[1]]$`_id`
#> [1] "3"
#>
#> $docs[[1]]$`_version`
#> [1] 1
#>
#> $docs[[1]]$found
#> [1] TRUE
#>
#> $docs[[1]]$`_source`
#> $docs[[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0107756"
#>
#> $docs[[1]]$`_source`$title
#> [1] "The Effect of S-Adenosylmethionine on Cognitive Performance in Mice: An Animal Model Meta-Analysis"
#>
#>
#>
#> $docs[[2]]
#> $docs[[2]]$`_index`
#> [1] "plos"
#>
#> $docs[[2]]$`_type`
#> [1] "article"
#>
#> $docs[[2]]$`_id`
#> [1] "4"
#>
#> $docs[[2]]$`_version`
#> [1] 1
#>
#> $docs[[2]]$found
#> [1] TRUE
#>
#> $docs[[2]]$`_source`
#> $docs[[2]]$`_source`$id
#> [1] "10.1371/journal.pone.0107758"
#>
#> $docs[[2]]$`_source`$title
#> [1] "Lactobacilli Inactivate Chlamydia trachomatis through Lactic Acid but Not H2O2"

Different indeces, types, and ids

docs_mget(index_type_id=list(c("plos","article",1), c("gbif","record",1)))$docs[[1]]
#> $`_index`
#> [1] "plos"
#>
#> $`_type`
#> [1] "article"
#>
#> $`_id`
#> [1] "1"
#>
#> $`_version`
#> [1] 1
#>
#> $found
#> [1] TRUE
#>
#> $`_source`
#> $`_source`$id
#> [1] "10.1371/journal.pone.0098602"
#>
#> $`_source`$title
#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"

Raw JSON data

You can optionally get back raw json from Search(), docs_get(), and docs_mget() setting parameter raw=TRUE.

For example:

(out <- docs_mget(index="plos", type="article", id=5:6, raw=TRUE))
#> [1] "{\"docs\":[{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"5\",\"_version\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0085123\",\"title\":\"MiR-21 Is under Control of STAT5 but Is Dispensable for Mammary Development and Lactation\"}},{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"6\",\"_version\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0098600\",\"title\":\"Correction: Designing Mixed Species Tree Plantations for the Tropics: Balancing Ecological Attributes of Species with Landholder Preferences in the Philippines\"}}]}"
#> attr(,"class")
#> [1] "elastic_mget"

Then parse

jsonlite::fromJSON(out)
#> $docs
#>   _index   _type _id _version found                   _source.id
#> 1   plos article   5        1  TRUE 10.1371/journal.pone.0085123
#> 2   plos article   6        1  TRUE 10.1371/journal.pone.0098600
#>                                                                                                                                                     _source.title
#> 1                                                                       MiR-21 Is under Control of STAT5 but Is Dispensable for Mammary Development and Lactation
#> 2 Correction: Designing Mixed Species Tree Plantations for the Tropics: Balancing Ecological Attributes of Species with Landholder Preferences in the Philippines

Citing

Scott Chamberlain (2016). elastic: General Purpose Interface to Elasticsearch. R package version 0.6.0. http://cran.rstudio.com/package=elastic

License and bugs

Back to top

comments powered by Disqus