Elasticsearch+logstash+jdbc

logstash configuration:

output {
   elasticsearch {
      hosts => ["192.168.8.8:9200"]
      index => "jdbc_index"
      document_id => "%{id}"
   }

   stdout {
      codec => json_lines
   }
}

it is configured to output in two ways. Why is there only one record for displaying ES using kibana

clipboard.png
stdout:

clipboard.png

has been doing this for a long time. Do any seniors know why?

configuration file code:

input {
   stdin {
   }
   jdbc {
      jdbc_connection_string => "jdbc:mysql://192.168.12.222:3306/main"
      jdbc_user => "admin"
      jdbc_password => "123456"
      jdbc_driver_library => "/usr/local/logstash/mysql-connector-java-5.1.3.jar"
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      statement_filepath => "/usr/local/logstash/config/jdbc.sql"
      -sharpstatement => "select inquiryId, inquiryNo, carTypeId, orgId, `status`, createTime, publishTime, failTime from qp_inquiry limit 500"
      schedule => "* * * * *"
      type => "jdbc_type"
      lowercase_column_names => "false"
   }
}

filter {
   json {
      source => "message"
      remove_field => ["message"]
   }
}

output {
   elasticsearch {
      hosts => ["192.168.8.8:9200"]
      index => "jdbc_index"
      document_id => "%{id}"
   }

   stdout {
      codec => json_lines
   }
}

sql statement:

SELECT
*
FROM
 (
  SELECT
   inquiryId,
   inquiryNo,
   carTypeId,
   orgId,
   `status`,
   createTime,
   publishTime,
   failTime
  FROM
   qp_inquiry
  LIMIT 50
 ) as `t1`
LIMIT 50000 OFFSET 0

see that there is no id in your original field. You should specify a field that exists

.
Menu