使用sls-logger来处理apisix access log #

Created by: xiaowei Luo Tags: SLS Created time: January 10, 2024 11:03 AM

根据加工原理，数据加工任务的总体速度取决于源Shard的数量、用户配置的规则逻辑和规则复杂度。一般可以按照每Shard处理1MB/s（压缩前）流量规划，也就是大约85 GB每天每Shard规划。例如：源Logstore的数据写入速度是每天1 TB，那么需要分裂源Logstore的Shard数量为1024GB/85=12个. 需要源和目的shards 5 * 12 = 60

Untitled

通过 sls-logger 写入 sls的 access log 需要额外处理 1.

# 样例
{"_program_":"apisix","_priority_":"46","_facility_":"5","_severity_":"6","_unixtimestamp_":"1704857318010000000","_hostname_":"edu-stress.seewo.com","content":"{\"time_local\":\"10\\/Jan\\/2024:11:28:38 +0800\",\"topic\":\"apisix-kafka-logger\",\"traceId\":\"e3607c2fcfb999dbce460d47ea15f065\",\"body_bytes_sent\":\"23999\",\"route_id\":\"782e2b7a89914795b8f9bc9d_80f732ff2ac54d3c9387a71f\",\"remote_user\":\"-\",\"http_user_agent\":\"PerfomanceTest\",\"remote_addr\":\"10.226.0.192\",\"upstream_time\":0.063,\"server_protocol\":\"HTTP\\/1.1\",\"forward\":\"-\",\"pod_ip\":\"172.26.0.246\",\"request_time\":0.063,\"request\":\"\\/open\\/api\\/v1\\/courseware\\/2\\/detail?shareLinkUid=&accessCode=bh_Keo4bMngTkTt7hIA0Qq8e46rnN1Tir-Dtnpjc3ahhU8g_3uebjzMRLYnuITydY9tvlbpT2MN6t8&cid=&turnMp4=true&expireSeconds=30&w=964&h=542&turnWebp=true\",\"method\":\"GET\",\"status\":200,\"domain\":\"edu-stress.seewo.com\",\"origin_request\":\"GET \\/open\\/api\\/v1\\/courseware\\/2\\/detail?shareLinkUid=&accessCode=bh_Keo4bMngTkTt7hIA0Qq8e46rnN1Tir-Dtnpjc3ahhU8g_3uebjzMRLYnuITydY9tvlbpT2MN6t8&cid=&turnMp4=true&expireSeconds=30&w=964&h=542&turnWebp=true HTTP\\/1.1\",\"request_length\":747,\"http_referer\":\"-\",\"upstream_status\":\"200\",\"upstream\":\"10.226.0.38:8080\"}","_process_id_":"78","__pack_meta__":"8|MTcwMjQwOTAyNTY2NjcyNDU5OQ==|2366|2047","__topic__":"syslog-forwarder","__source__":"edu-stress.seewo.com","__tag__:__pack_id__":"3E5E43B99C0CF8DC-376","__tag__:__receive_time__":"1704857319","__time__":"1704857318"}

他的数据是包在content里面的，因此需要将content解析出来存到另外一个logstore

e_compose(
    e_json("content", depth=1), e_drop_fields("content", "origin_request"),
)

apisix 配置启用 sls-logger配置

{
  "host": "cn-hangzhou-intranet.log.aliyuncs.com",
  "name": "sls logger",
  "port": 10009,
  "project": "ccloud-apisix-log",
  "logstore": "ccloud-apisix-log",
  "access_key_id": "LTAI5tFK4fbWYYyQdWQ3kr67",
  "access_key_secret": "bIhvqAPIBcL8YIkqkFGPXHC1oxE60j"
}

针对 200 集群启用采样

通过流水线和接口更新插件配置

UPDATE `t_ingress_rules` SET `c_plugin_config` = CAST('{"host": "cn-hangzhou-intranet.log.aliyuncs.com", "name": "sls logger", "port": 10009, "project": "ccloud-apisix-log", "logstore": "ccloud-apisix-log", "access_key_id": "LTAI5tFK4fbWYYyQdWQ3kr67", "access_key_secret": "bIhvqAPIBcL8YIkqkFGPXHC1oxE60j"}' AS JSON) WHERE `c_id` = 1898;

curl -X PUT -H 'Authorization: 195afaa0-568f-11ed-96a1-0673683df9be' -H 'Content-Type: application/json' 'http://ccloud.cvte.com/cloud/api/v1/open/ingress-rule/global/200/benchmark-dep'