Google PubSub Inputプラグインを使用したLogstash「エラー：証明書の確認に失敗しました」

Question

Google PubSub Pluginを機能させるのに苦労しています。すべての設定手順（トピックの作成、専用サービスアカウントの作成など）構成は次のようになります。私は&実行それを構築する際に

input { 
    google_pubsub { 
     project_id => "secret" 
     topic => "fw-services-logs" 
     subscription => "fw-logstash-sub" 
     json_key_file => "/usr/share/logstash/service_account_key.json" 
    } 
} 

output { stdout { codec => rubydebug } } 


# Works: 
# input { stdin { } } 
# output { stdout { codec => rubydebug } } 

はしかし、私は次のような出力が得られます。

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties 
[2017-10-27T11:39:46,455][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} 
[2017-10-27T11:39:46,462][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} 
[2017-10-27T11:39:46,694][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.6.3-java/modules/arcsight/configuration"} 
[2017-10-27T11:39:46,714][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} 
[2017-10-27T11:39:46,720][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} 
[2017-10-27T11:39:46,795][INFO ][logstash.agent   ] No persistent UUID file found. Generating new UUID {:uuid=>"a144c8d2-12a9-4a15-a629-59910dda9295", :path=>"/usr/share/logstash/data/uuid"} 
[2017-10-27T11:39:48,114][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@elasticsearch:9200/]}} 
[2017-10-27T11:39:48,118][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@elasticsearch:9200/, :path=>"/"} 
[2017-10-27T11:39:48,343][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"} 
[2017-10-27T11:39:48,345][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]} 
[2017-10-27T11:39:48,346][INFO ][logstash.pipeline  ] Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2} 
[2017-10-27T11:39:48,389][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@elasticsearch:9200/]}} 
[2017-10-27T11:39:48,390][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@elasticsearch:9200/, :path=>"/"} 
[2017-10-27T11:39:48,404][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"} 
[2017-10-27T11:39:48,435][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://logstash_system:xxxxxx@elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} 
[2017-10-27T11:39:48,445][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} 
[2017-10-27T11:39:48,446][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server 
[2017-10-27T11:39:48,486][INFO ][logstash.pipeline  ] Pipeline .monitoring-logstash started 
[2017-10-27T11:39:48,536][INFO ][logstash.pipeline  ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250} 
[2017-10-27T11:39:50,049][INFO ][logstash.inputs.googlepubsub] Client authorizataion with JSON key ready 
[2017-10-27T11:39:50,051][INFO ][logstash.pipeline  ] Pipeline main started 
[2017-10-27T11:39:50,235][INFO ][logstash.agent   ] Successfully started Logstash API endpoint {:port=>9600} 
[2017-10-27T11:39:50,875][ERROR][logstash.pipeline  ] A plugin had an unrecoverable error. Will restart this plugin. 
    Plugin: <LogStash::Inputs::GooglePubSub project_id=>"secret", topic=>"fw-services-logs", subscription=>"fw-logstash-sub", json_key_file=>"/usr/share/logstash/service_account_key.json", id=>"23998ce4135823320ff87ea3bf3ca73094144ca9-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_41007486-427e-4323-99fd-9375470f239e", enable_metric=>true, charset=>"UTF-8">, max_messages=>5> 
    Error: certificate verify failed 

私はデバッグログレベルで実行すると、私はそれがどこかにファラデーでクラッシュすることを参照してください。

[2017-10-27T12:42:01,538][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. 
    Plugin: <LogStash::...> 
    Error: certificate verify failed 
    Exception: Faraday::SSLError 
    Stack: org 

質問：

• 私が失敗したため、エラーを確認し、証明書のプラグインのクラッシュを修正することができますどのように任意のアイデア？
• ElasticSearchに関するこれらの警告/情報ログは正常ですか？ AFAIK ESを使う必要はありませんか？

私が使用してドッキングウィンドウのファイル：

FROM docker.elastic.co/logstash/logstash:5.6.3 

ADD service_account_key.json /usr/share/logstash/service_account_key.json 

# https://www.elastic.co/guide/en/logstash/current/docker.html 
RUN rm -f /usr/share/logstash/pipeline/logstash.conf 
ADD ./pipeline/ /usr/share/logstash/pipeline/ 

RUN logstash-plugin install logstash-input-google_pubsub 

を私は構築したいと、それを実行します。

docker build -t logstash -f logstash.docker --no-cache . 
docker run -it logstash 

Answer 1

が明示的にSSL_CERT_FILEは、問題を修正し設定します。

# see https://github.com/google/google-api-ruby-client/issues/253 
ENV SSL_CERT_FILE /usr/share/logstash/logstash-core/lib/logstash/certs/cacert.pem 

デフォルトでは、Logstashは監視情報をElasticsearchに記録しようとします。 あなたが、私の場合のようにそれが起こるしたくない場合はあなたがそれを無効にすることができます。問題のこれらの種類をデバッグするため

ENV XPACK_MONITORING_ENABLED false 

それはdebugにLogstashログレベルを設定するには便利です：

ENV LOG_LEVEL debug