Logstashは、私がURLからクエリパラメータを抽出しようとしているURL

Question

から配列インデックスの解析に失敗しました。

127.0.0.1 - - [09/May/2016:09:32:19 +0200] "GET /ps?attrib[vendor][]=GOK&attrib[vendor][0]=GOK HTTP/1.1" 200 12049 "-" "-" 

（予想通り）attribの最初の出現は、ハッシュを生成します。私はパースだログファイルから厄介行は、次のようになります。しかし、第2の出現は、例外につながる：

IndexError: string not matched 
      []= at org/jruby/RubyString.java:3910 
      set at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.3-java/lib/logstash/util/accessors.rb:64 
      []= at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.3-java/lib/logstash/event.rb:136 
     filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-kv-2.1.0/lib/logstash/filters/kv.rb:287 
      each at org/jruby/RubyHash.java:1342 
     filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-kv-2.1.0/lib/logstash/filters/kv.rb:287 
    multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/filters/base.rb:151 
      each at org/jruby/RubyArray.java:1613 
    multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/filters/base.rb:148 
    filter_func at (eval):189 
    filter_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:267 
      each at org/jruby/RubyArray.java:1613 
     inject at org/jruby/RubyEnumerable.java:852 
    filter_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:265 
    worker_loop at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:223 
    start_workers at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:201 

私はインデックスが実際に整数でありながらlogstashは、文字列としてURLから配列のインデックスを解釈するためだと思います。
は異なるのconfigsをグーグルと試みるの日後、私は行き止まりに来ています。どのようにこの作品を作るためのアイデア？デバッグの目的のために

：

logstashコンフィグ

input { 
    file { 
    path => "/var/log/apache2/some.log" 
    start_position => "beginning" 
    sincedb_path => "/dev/null" 
    } 
} 

filter { 
    grok { 
    match => { 
     "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth}\s?(%{NUMBER:seconds:int}\/%{NUMBER:microseconds:int})? \[%{HTTPDATE:timestamp}\] "%{WORD:verb} (%{WORD:schema}:)?[\S]+/(%{DATA:endpoint})\?%{DATA:query_string} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}(\s{1}(?:%{HOSTNAME:backend_used}|-) (?:%{NUMBER:backend_time_seconds:float}|-)s)?' 
    } 
    } 

    urldecode { 
    field => "query_string" 
    charset => "ISO-8859-1" 
    } 

    kv { 
    field_split => "&" 
    source => "query_string" 
    recursive => true 
    allow_duplicate_values => false 
    } 

    date { 
    match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] 
    locale => en 
    } 

    geoip { 
    source => "clientip" 
    } 

    useragent { 
    source => "agent" 
    target => "useragent" 
    } 
} 

output { 
    stdout { 
    codec => json 
    } 
} 

カスタムダイナミックテンプレート

{ 
    "template": "apache_elk_example", 
    "settings": { 
    "index.refresh_interval": "5s" 
    }, 
    "mappings": { 
    "_default_": { 
     "numeric_detection" : true, 
     "dynamic_templates": [ 
      { 
       "message_field": { 
       "mapping": { 
        "index": "analyzed", 
        "omit_norms": true, 
        "type": "string" 
       }, 
       "match_mapping_type": "string", 
       "match": "message" 
       } 
      }, 
      { 
       "string_fields": { 
       "mapping": { 
        "index": "analyzed", 
        "omit_norms": true, 
        "type": "string", 
        "dynamic": true, 
        "fields": { 
         "raw": { 
          "index": "not_analyzed", 
          "ignore_above": 256, 
          "type": "string" 
         } 
        } 
       }, 
       "match_mapping_type": "string", 
       "match": "*" 
       } 
      } 
     ], 
     "properties": { 
      "geoip": { 
       "dynamic": true, 
       "properties": { 
       "location": { 
        "type": "geo_point" 
       } 
       }, 
       "type": "object" 
      }, 
      "@version": { 
       "index": "not_analyzed", 
       "type": "string" 
      } 
     }, 
     "_all": { 
      "enabled": true 
     } 
    } 
    } 
} 

と attrib[vendor][0]=GOK以来

Answer 1

は、あなたがちょうどいいkvフィルタの前の数字のインデックスを削除すると考えられてきた、かなり多くのPHPのために意味的に等しいですか？以下のような何か：

# Remove numeric indices from arrays, otherwise the kv filter will choke, eg: 
    # attrib[vendor][0]=GOK becomes attrib[vendor][]=GOK 
    mutate { 
    gsub => [ 
     "query_string", "\[\d+\]", "[]" 
    ] 
    } 

    kv { 
    ... 
    }