申玉寶+hbase在大搜車金融業(yè)務(wù)中的應(yīng)用實(shí)踐-22頁

《申玉寶+hbase在大搜車金融業(yè)務(wù)中的應(yīng)用實(shí)踐-22頁》由會(huì)員上傳分享，免費(fèi)在線閱讀，更多相關(guān)內(nèi)容在行業(yè)資料-天天文庫。

1、HBase在大搜車金融業(yè)務(wù)中的應(yīng)用實(shí)踐高級(jí)數(shù)據(jù)架構(gòu)師申玉寶1234業(yè)務(wù)介紹數(shù)據(jù)大屏實(shí)踐GPS風(fēng)控實(shí)踐流式數(shù)據(jù)統(tǒng)計(jì)數(shù)據(jù)大屏實(shí)踐車輛軌跡大屏Table:數(shù)據(jù)倉庫(Hive)HBase(Phoenix)GPS.TRACESROWKEY:reverse(DEVICE_ID)+TS消息隊(duì)列COLUMNS:RK,LON,LAT,VENDOR_ID,CITY_ID,ACC...GPS上報(bào)網(wǎng)關(guān)數(shù)據(jù)網(wǎng)關(guān)INDEX:ONGPS.TRACES(GPS_TIME,CITY_ID,VENDOR_ID)INCLUDE(LON,LAT)車載GPS設(shè)備軌跡大屏大屏性能優(yōu)化–

2、WebsocketF2EServerHBaseWebSocketConnect?數(shù)據(jù)分片,減少首次請(qǐng)求耗時(shí)Query01:00Send01:0001:00Query02:00?渲染線程與數(shù)據(jù)請(qǐng)求線程獨(dú)立,保障后續(xù)流暢Send02:00Query03:00Send32:00?使用WebSocket減少建立HTTP請(qǐng)求耗時(shí)02:0003:00實(shí)時(shí)成交大屏DIM基礎(chǔ)數(shù)據(jù)庫Table:(Mysql)TRD.TGC_ORDERS埋點(diǎn)日志MQROWKEY:reverse(ORDER_ID)MQ精衛(wèi)COLUMNS:ConsumerPAY_DATE,STOR

3、E_ID,STORE_TYPE,STORE_AREA,BUYER_ID,CAR_ID...HBase(Phoenix)INDEX:ONGPS.TGC_ORDERS(PAY_DATE,STORE_TYPE,CAR_MODEL,STORE_AREA)INCLUDE(STORE_NAME,AMOUNT)數(shù)據(jù)網(wǎng)關(guān)成交大屏GPS風(fēng)控實(shí)踐GPS風(fēng)控-業(yè)務(wù)系統(tǒng)GPS風(fēng)控-業(yè)務(wù)架構(gòu)軌跡監(jiān)控實(shí)時(shí)軌跡廠商設(shè)備注冊轉(zhuǎn)發(fā)上報(bào)電子圍欄狀態(tài)維護(hù)風(fēng)控特征模型預(yù)警MQHBase停留點(diǎn)分析提取&里程糾偏后臺(tái)(ETL)規(guī)則引擎聚集分析設(shè)備直連GPS上報(bào)網(wǎng)關(guān)…..聚集分析-Geo

4、Hash?對(duì)地球進(jìn)行二維平面化,不停進(jìn)行32等分?Base32編碼字符串,每個(gè)字符由5bit組成?每個(gè)字符串代表地球的一個(gè)區(qū)域聚集分析-計(jì)算過程上報(bào)GPS數(shù)據(jù)到數(shù)據(jù)網(wǎng)關(guān)發(fā)送貸后管理人員跟進(jìn)數(shù)據(jù)糾偏規(guī)則引擎聚合其他特征入庫HBase根據(jù)Left(GeoHash,6)聚合,取出聚合點(diǎn)根據(jù)最近軌跡分析出每輛車停留點(diǎn)(定時(shí))根據(jù)車輛停留點(diǎn),生成GeoHash聚集分析-數(shù)據(jù)存儲(chǔ)原始軌跡聚集點(diǎn)支持:按設(shè)備,時(shí)間維度查詢詳細(xì)軌跡支持:按區(qū)域,時(shí)間維度查詢聚集數(shù)據(jù)Table:Table:GPS.TRACESGPS.GATHERPOINTSROWKEY:ROW

5、KEY:reverse(DEVICE_ID)+TSreverse(TS)+reverse(left(GeoHash,6))COLUMNS:COLUMNS:DEVICEIDE,TS,LON,LAT,VENDOR_ID,CITY_ID,ACC...TS,GEOHASH,DEVICE_ID…流式數(shù)據(jù)統(tǒng)計(jì)流式數(shù)據(jù)統(tǒng)計(jì)-業(yè)務(wù)場景業(yè)務(wù)特點(diǎn):?實(shí)時(shí)數(shù)據(jù)統(tǒng)計(jì)(<10s)?數(shù)據(jù)量較大(1m~100m)?有并發(fā)要求(~100qps)?查詢維度多且易變流式數(shù)據(jù)統(tǒng)計(jì)–數(shù)據(jù)流基礎(chǔ)維表業(yè)務(wù)ARDS業(yè)務(wù)數(shù)業(yè)務(wù)B據(jù)庫Phoenix數(shù)據(jù)RDS精衛(wèi)CacheMID網(wǎng)關(guān)業(yè)務(wù)CMQ

6、埋點(diǎn)數(shù)Phoenix據(jù)DMLog...流式數(shù)據(jù)統(tǒng)計(jì)–數(shù)據(jù)合流訂單ID訂單金額訂單ID車輛品牌10001200,00010001寶馬訂單表訂單-車輛表訂單-店鋪表(主流)(支流)(支流)10002250,00010002奧迪10003320,00010003奔馳10004600,00010004特斯拉數(shù)據(jù)緩存數(shù)據(jù)緩存數(shù)據(jù)緩存訂單ID店鋪類型100014S店10002二手車商10003新車二網(wǎng)100044S店合流檢測訂單ID店鋪類型車輛品牌訂單金額數(shù)據(jù)整合100014S店寶馬200,00010002二手車商奧迪250,00010003新車二網(wǎng)奔馳

7、320,000100044S店特斯拉600,000輸出Phoenix流式數(shù)據(jù)統(tǒng)計(jì)–性能測試并發(fā)TPSRT(ms)阿里云HBase:10139.5671.21Master(2C4G)+CORE(4C8G)*220197.7999.82數(shù)據(jù)量:~100million30194.17159.36實(shí)驗(yàn)條件:40209.45192.66?線上流量復(fù)制50220.77225.47?包含大量Group查詢?短路相關(guān)緩存60221.29267.71?全Phoenix索引覆蓋70208.50339.3180220.52358.2590220.64415.061

8、00223.48439.98*本數(shù)據(jù)僅為大搜車內(nèi)部測試,可能受測試條件影響,不代表官方標(biāo)準(zhǔn)流式數(shù)據(jù)統(tǒng)計(jì)–服務(wù)監(jiān)控