Elasticsearch json aggregation search - Codes Helper - Programming Question Answer

Elasticsearch json aggregation search

now there is a need to count user operations, because there must be a large amount of data lost in mysql, so we are going to use elasticsearch. At present, there is a problem. Post the structure directly

.

{
  "_index": "test_vpn_operation-2018-04-08",
  "_type": "vpn_operation",
  "_id": "AWKiw45UfMhbLGkLP-v3",
  "_score": null,
  "_source": {
    "app_version": "2.0.0",
    "dateline": 1523149062,
    "channel": "pc",
    "edition": "jichu",
    "message": "{\"date_time\":\"2018-04-08T08:59:07\",\"account_id\":447189,\"operation\":[{\"account_id\":\"447189\",\"operation_id\":1,\"timestamp\":1523149062},{\"account_id\":\"447189\",\"operation_id\":1,\"timestamp\":1523149063}],\"channel\":\"pc\",\"edition\":\"jichu\",\"app_version\":\"2.0.0\",\"dateline\":1523149062}",
    "type": "vpn_operation",
    "path": "/data/wwwroot/vpnApi/elklog/vpn_operation_record_20180408",
    "@timestamp": "2018-04-08T00:59:07.722Z",
    "account_id": 447189,
    "date_time": "2018-04-08T08:59:07",
    "@version": "1",
    "host": "JG-otter",
    "operation": [
      {
        "operation_id": 1,
        "account_id": "447189",
        "timestamp": 1523149062
      },
      {
        "operation_id": 1,
        "account_id": "447189",
        "timestamp": 1523149063
      }
    ]
  },
  "fields": {
    "date_time": [
      1523177947000
    ],
    "@timestamp": [
      1523149147722
    ]
  },
  "highlight": {
    "message": [
      "{\"date_time\":\"2018-04-08T08:59:@kibana-highlighted-field@07@/kibana-highlighted-field@\",\"account_id\":447189,\"operation\":[{\"account_id\":\"447189\",\"operation_id\":1,\"timestamp\":1523149062},{\"account_id\":\"447189\",\"operation_id\":1,\"timestamp\":1523149063}],\"channel\":\"pc\",\"edition\":\"jichu\",\"app_version\":\"2.0.0\",\"dateline\":1523149062}"
    ]
  },
  "sort": [
    1523149147722
  ]
}

originally intended to implement mysql similar to group by statistical operations by aggregating operation.operation_id, but found that the data was not accurate. Because the operation_id field does not exist under operation, is there any way to achieve similar requirements, or how I need to adjust the structure to achieve the requirements

Mar.02,2021

guess 1: when you set mapping, you use an index of object type on the object array, which should be the default strategy for nested documents. If an index of type object is used, the array of objects will be flattened. See the following three official documents:

ide/cn/elasticsearch/guide/cn/complex-core-fields.html-sharpobject-arrays" rel=" nofollow noreferrer "> an array of internal objects
ide/cn/elasticsearch/guide/cn/nested-objects.html" rel= "nofollow noreferrer" > nested objects
ide/cn/elasticsearch/guide/cn/nested-mapping.html" rel= "nofollow noreferrer" > nested object mapping

guess 2: whether some of your documents do not have the field operation.operation_id , resulting in inaccurate results. At this point, you can set the default value for this field. Official document: ide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html-sharp_missing_value_9" rel=" nofollow noreferrer "> missing value

guess 3: whether the operation.operation_id field has multiple types in all documents. For example, at first I saved int, then I saved string.. I do it against the norm because my upstream data source is the log, and field changes in the log are common. My practice is to check only the changed part of the index, because: the mapping change rule takes effect when the new index is established. For example, the index-2018-03-02 index rule will not take effect until it is changed to mapping, on March 1. At this time, a cross-index query, such as a query for the whole 3 months, will go wrong. You can query from March 2 to March 31 at this time.

Previous: How to get the timing of webpack asynchronous loading?

Next: When using quartz, I want to execute jobDetail at an interval of 6000 each time. How can I modify it?

How do elasticsearch nested objects split query results?
there are multiple items under an order. The order object is stored in the form of nested, and an order is a piece of data. order 1-> Commodity A, Commodity B, Commodity C, Commodity D order 2-> Commodity A, Commodity B, Commodity C, Commodity D. or...

Aggregate query elasticsearch

Feb.28,2021
Why does the type of geo_point not take effect if a field in the logstash log is set?
the architecture is simple: filebeat collects nginx logs, output to logstash logstash format and then output to elasticsearch There is nothing to say about the configuration of filebeat but to send the access.log of nginx directly to logstash ...

Filebeat logstash elk elasticsearch kibana

Feb.28,2021
Questions about the Chapter "Mapping and Analysis" of elasticsearch Chinese documents
document connection here the above example cannot be realized at all I try the following: POST gb tweet _bulk { "index": { "_id": 1 }} { "name" : "MySQL", "user_id" : "15", "twee...

Elasticsearch elastic-search

Mar.03,2021
After Elasticsearch, uses ik participle, the result of the aggregate query is lowercase letters?
When the value of the field is AA , the result of the aggregate query is how does aa , get the original value? make the following query: POST 113 _search { "query": { "query_string": { "query": "J...

Elasticsearch

Mar.03,2021
10 pieces of data sampled by elasticsearch
such as the title. how does elasticsearch sample 10 items from millions of data, which ensure the ranking of correlation and try not to repeat the search with the same conditions in a small number of times. ...

Elasticsearch elastic-search

Mar.03,2021
Spring Cloud distributed link tracking sleuth+rabbitMQ+ElasticSearch+Zipkin integration failed
phenomenon: the service cannot be found using ElasticSearch as the storage method. There is no problem with using MySQL as the storage method before, and the service can be found quickly now the console does not report an error properties file...

Zipkin rabbitmq elasticsearch springcloud

Mar.10,2021
How does elasticsearch implement such a search?
using elasticsearch as a full-text search engine, we now encounter a problem. The index data comes from the commodity table good, assumes that the search keyword "tractor " can find 100 items of data, belonging to five enterprises, each of which upload...

Full-text-search elasticsearch

Mar.10,2021
Elasticsearch is reasonable about how to design index and type.
now there are several completely different statistical data structures for a city, which are obtained from parsing tables. If you put dozens of type, under an index in the mapping of ES, there are probably dozens of type, under different Index or all und...

Index mapping elasticsearch

Mar.11,2021
What is the syntax of kibana filter mismatch?
I want to drop the url field in kibana visualize, Filter, which ends with js. How do I write query dsl?? I can t write it in the following way { "query": { "bool": { "must_not": [ { "mat...

Elasticsearch kibana

Mar.13,2021
Query+aggs usage problems in EalsticSearch
version: es 2.4.6 An index with string type field named seller I want to aggregate part of the data by adding query to satisfy query . As shown below, only query seller=xxx data, is it written correctly? seller aaa,aaa_bbb sellerccc-dd...

Elasticsearch

Mar.13,2021
Logstash-6.2.2 startup failed
1.ES 9200 port started 2.logstashhost192.168.33.10 what is the cause of the error? How to solve it? ...

Java php kibana logstash elasticsearch

Mar.14,2021
How to check whether kibana and elsasticsearch-head services are started
the kibana and elasticsearch-head corresponding to the higher version of elasticsearch are developed by nodejs, and the startup is also started through the command npm run start of nodejs but after startup, how to determine whether the service is star...

Elasticsearch kibana

Mar.18,2021
Elasticsearch uses spring data to aggregate queries. Filter condition does not take effect.
** * * @return list * public List<UserResponseVO> listConditionsAggregations() { TermsAggregationBuilder builder = AggregationBuilders.terms("interests").field("interests"); SearchQuery query = new NativeSearchQuery...

Elasticsearch spring java

Mar.18,2021
Elasticsearch uses spring data to aggregate queries. Filter condition does not take effect.
** * * @return list * public List<UserResponseVO> listConditionsAggregations() { TermsAggregationBuilder builder = AggregationBuilders.terms("interests").field("interests"); SearchQuery query = new NativeSearchQuery...

Java spring elasticsearch

Mar.18,2021
In ES, how to use templates for all fields under a field, and not for fields outside the field.
now is a scenario where there are 50 fields under field A, and these fields are of the same type as the parser. You don t want to specify one by one manually. You want to set the template. but I m afraid that setting a template will affect all field...

Elk elasticsearch

Mar.19,2021
Unable to set password and login after installing x-pack
has installed elasticsearch, version 6.2.4, and started normally has installed kibana, version 6.2.4 and started normally x-pack version is installed according to the official method, 6.2.4 elasticsearch and kibana start up normally however, d...

Elasticsearch kibana

Mar.20,2021
How to implement Array containing query by es (elasticsearch)
es (elasticsearch) how to implement the array containing query index data mVIA: [1,2,3] index array NVIA: [1mem3lce4] query condition: a: [1p3L4] requires to include query, can find index data n, index data n is in the query condition, index data m ...

Elastic-search elasticsearch

Mar.21,2021
How many inputs of filebeat nodes can logstash hold?
excuse me, A 4-core 8G machine with logstash, deployed on it can probably support several log input nodes of filebeat. If there is no message queue as buffer, do you have any experience in this field? similarly, how many logstash nodes can a 4-core ...

Filebeat elasticsearch logstash elk

Mar.21,2021
ImportError: cannot import name InnerObjectWrapper
Python 2.7.12, elasticsearch 6.2.0 elasticsearch-dsl 6.1.0 from elasticsearch_dsl import InnerObjectWrapper Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name InnerObjectWrap...

Python2.7 elasticsearch

Mar.22,2021
How does elasticsearch read one-to-many, parent-child data structures at one time?
for example, the data structure is as follows: posts-- comments on posts-- comment users 3 layers. now you need to query a post. It is best to find the comments under the post and the user data under the comments. can a query be done? ...

Elasticsearch

Mar.22,2021

Menu