Hawlgallada IT — kalsoonida goobta (SRE) Qaabka tooska ah

Waxaad ku jirtaa kooxda deganaanshaha (on-call) ee adeeg shabakad: isu-dheelitirka culayska ka hor kooxda mashiinnada, kaash ka hor kaydka xogta, iyo bartilmaameedka helitaanka 99.9 %. Daqiiqad kasta moodelku wuxuu xisaabiyaa daahitaanka safka, waqtiga xadka, garaacidda kaashka iyo culayska kaydka xogta iyadoo la adeegsanayo qaacidooyin buug-tacliineed — iyo waxa go'aamadaadu ku kacaan.

Waxa aad baran doonto

Jile

Waqti 0 min
Codsiyo 1125 · Heerka khaladaadka 0.00% · Daahitaanka p99 342 ms · Mashiinno adeegaya 10 (+0) · Heerka garaacidda kaashka 77% · Isticmaalka kaydka xogta 19% · Miisaaniyadda khaladaadka haray (30 maalmood) 50.0%⇉1125 codsi/s▶▶▶▶▶▶▶▶▶▶······························▶ 10 · ⚙ 0 · 52% · p99 342 msIsticmaalka kooxda mashiinnada 52%Heerka garaacidda kaashka 77%Isticmaalka kaydka xogta 19%⚠ 0.00% · 🔥 0.0×50%$ 4.00/h · Σ $0.00
  • Mashiin adeegaya
  • Mashiin bilaabmaya
  • Mashiin ku jira dhismaha xun
  • Meel bannaan
  • Codsiyo soo gelaya

Xakamayaal

Dabaqa kooxda mashiinnada. Kordhintiisu wuxuu isla markiiba bilaabaa mashiinno — weli waxay u baahan yihiin daahitaanka bilowga ka hor intaan adeegin.

Raacitaanka bartilmaameedka isticmaalka; ma jiro balaadhin cusub inta mashiinnadu weli bilaabmayaan. Dansan = si sax ah ugu yaraan.

Hoose = boos badan iyo kharash badan. Isticmaalka la cabbiray ma dhaafi karo 100 %, sidaas darteed koox buuxsan waxay kortaa tallaabo-tallaabo oo keliya.

TTL dheer = garaacid badan, laakiin jawaabuhu way da'aysan karaan (da'da celceliska ≈ TTL/2).

Waxay raraan furayaasha inta badan la isticmaalo 6 daqiiqo (+12 % kaydka kaashka daqiiqad kasta) qiimaha 600 weydiin oo dheeraad ah oo kaydka xogta ah/s.

Qayb ka mid ah 30 % mudnaanta hoose (hore-u-rarid, xidhmo, gurguurayaasha raadinta) oo lagu diido isu-dheelitirka culayska "isku day mar dambe".

Sifada culusi waxay ku dartaa 20 ms CPU iyo hal weydiin kaydka xogta codsi kasta. Dansan = hoos u dhac xakamaysan.

Dib ayuu u geliyaa dhismihii ugu dambeeyay ee wanaagsanaa koox cusub (5 daqiiqo, laba jeer la bixiyo), kadibna taraafikada ayuu u beddelaa. Mar kale taabashadu waxay dib u bilaabaysaa diyaarinta; haddii aanay jirin dhis xun oo nool, waxay ku kacaysaa lacag oo kaliya.

Tilmaamayaal

Heerka khaladaadka
0.00%
caadi
Daahitaanka p99
342ms
caadi
Miisaaniyadda khaladaadka haray (30 maalmood)
50.0%
caadi
Isticmaalka kooxda mashiinnada
52%
caadi
Xawaaraha gubashada (1 saac)0.0 ×
Codsiyo1125 req/s
Mashiinno adeegaya10
Mashiinno bilaabmaya0
Heerka garaacidda kaashka77 %
Isticmaalka kaydka xogta19 %
Taraafikada la diiday0 %
Kharashka kooxda mashiinnada4.00 $/h
Kharashka ilaa hadda0.00 $
Da'da celceliska ee jawaabaha kaashka30 s
Taraafikada dhismaha xun0 %
Talooyin la heli karo100 %

Isbeddel

Heerka khaladaadka: — %20.000.00

Xaaladaha dhibaatada

Heerka 1 · Sii deyn xun

Dhis cusub ayaa la sii daayaa 09:10. Bishu horeyba way adkayd: kaliya 20 % miisaaniyadda khaladaadka ayaa haray. Daqiiqado ka dib sii deynta, digniinta xawaaraha gubashadu way dhacdaa. Ilaali miisaaniyadda.

  • Miisaaniyadda khaladaadka haray dhammaadka ≥ 18.5 %
  • Heerka khaladaadka celceliska ≤ 0.65 % ka dib sii deynta
  • Kharashka kooxda mashiinnada ≤ $9.50

Heerka 2 · Dad-badan oo degdeg ah

Xiriirka adeegga ayaa si dhakhso ah u faafaya, goor subaxdan ka mid ah ayaa lagu filayaa kor-u-kac gaadiid — ma jiro qof ogaada goorta ama intee la eg yahay. Tusaalooyinka cusub maanta waxay qaataan 8 daqiiqo si ay u bilaabmaan. Marka uu yimaado, ku hay khaladaadka iyo dib-u-dhaca hoose adigoon lacag ku lumin awood aan shaqeyn.

  • Heerka khaladaadka celceliska ≤ 0.2 %
  • Daahitaanka p99 celceliska ≤ 400 ms
  • Taraafikada la diiday celceliska ≤ 5 %
  • Kharashka wadarta ≤ $21
  • Talooyin la heli karo ≥ 85 % waqtiga

Heerka 3 · Kaash qabow

Kor ugu sarraysa duhurka, qoraal dayactir ayaa kaashka oo dhan tirtira. Codsi kastaa hadda wuxuu aadaa kaydka xogta, kaas oo loo cabbiray heerka caadiga ah ee 85 % garaacid. Adeegga dib u soo celi adigoon kaydka xogta culayn xad-dhaaf ah.

  • Heerka khaladaadka celceliska ≤ 1.5 %
  • Isticmaalka kaydka xogta weligeed ha ka sarrayn 90 % daqiiqadda ugu horreysa ka dib
  • Da'da celceliska ee jawaabaha kaashka ≤ 90 s celcelis ahaan
  • Kharashka wadarta ≤ $9
  • Talooyin la heli karo ≥ 80 % waqtiga
  • Taraafikada la diiday celceliska ≤ 5 %

Aasaas — moodelka ka dambeeya tirooyinka

Xiriir kasta oo jilaha isticmaalo, iyo ilaha uu ka yimid. Mudnaanta lagu calaamadiyay malo-awaal waa hagaajin muunad ah.

Codsiyadu waxay raacaan qalooc maalinle ah oo lagu daray carqalado; tirada daqiiqaddii waa aan kala sooc lahayn (Poisson, qiyaas caadi ah) oo leh dhawr ruxrux.
λ(t) = base × (1 + 0.25·sin(2π(t + clock − 6 h)/24 h)) × surge(t), clock = time of day at the start (peak at 12:00); count/min ≈ N(60λ, √(60λ)) × (1 + N(0, 0.02))[6]Filasho: tirada shaqaalaha, wakhtiyada adeegga, awoodda kaydka xogta, cabbirka kaashka, xawaaraha buuxinta, qiimaha iyo heerka khaladaadka dhismaha xun waa qiyamyo tusaale ah oo loogu talagalay adeeg shabakad cabbir dhexe ah.
Erlang C: suurtagalnimada ah in codsi sugo shaqaale bannaan nidaamka M/M/N.
N = instances × 16 workers, a = λ·S; C(a, N) = B / (1 − (a/N)(1 − B)), B = Erlang B[3][6]
Dabada waqtiga sugitaanka: suurtagalnimada sugitaanka ka badan t waxay si jibaaran hoos u dhacdaa; codsiyada weli sugaya waqtiga xadka 2 s way fashilmaan. Awoodda ka sarraysa, dheeraadku wuu fashilmaa.
P(W > t) = C·e^(−(N/S − λ)·t); timeouts = P(W > 2 s); a ≥ N ⇒ failed share = 1 − N/a[3]
Daahitaanka p99 ee ka yimid quantile-yada waqtiga adeegga iyo sugitaanka.
p99 ≈ S·ln 100 + ln(C/0.01)/(N/S − λ) (service + waiting quantile, an approximation)[3][6]Qiyaas: isku-darka quantile-yada adeegga iyo sugitaanka ma aha p99-ka sax ah ee wadarkooda (waxaa laga yaabaa inuu in yar sarreeyo ama hooseeyo); safka waxaa loo daawadaa inuu deggan yahay daqiiqad kasta gudaheeda sababtoo ah codsiyadu waxay qaadanayaan milli-ilbiriqsi.
Sharciga Little: shaqaalaha mashquulka ah = xawaaraha imaatinka × waqtiga adeegga.
busy workers L = λ·S ⇒ utilization = λ·S / N[4]
Kaash TTL ah: codsiyo aan kala sooc lahayn, ka-wareeg kasta wuxuu bilaabaa muddo TTL ah oo codsiyadu garaacaan.
hit = warm × rT/(1 + rT), r = λ / 20,000 objects; mean age of a cached answer ≈ T/2[5]
Ka-wareegyada kaashku waxay culeeyaan kaydka xogta; daahitaankiisa safku wuxuu dhimaa codsi kasta oo taabta, taasoo sidoo kale buuxisa shaqaalaha codsiga.
DB load = λ·q·(1 − hit); query time = 5 ms/(1 − ρ_db) (≤ 250 ms); S = S_app + (1 − hit)·q·query time[6]Filasho: tirada shaqaalaha, wakhtiyada adeegga, awoodda kaydka xogta, cabbirka kaashka, xawaaraha buuxinta, qiimaha iyo heerka khaladaadka dhismaha xun waa qiyamyo tusaale ah oo loogu talagalay adeeg shabakad cabbir dhexe ah.
SLO iyo miisaaniyadda khaladaadka: xawaaraha gubashadu wuxuu sheegaa immisa jibaar ka dhakhso badan sida loo oggol yahay ayaa miisaaniyadda la isticmaalayaa.
budget = 1 − SLO = 0.1 %; burn = error rate / 0.1 %; Δbudget per min = burn / 43,200; burn (1 h) = mean error rate over the last 60 min / 0.1 % (window pre-filled with the opening minute)[1][2]
Isu-weyneeye toos ah oo bartilmaameed raac ah oo leh daahitaan bilow iyo waqti degdeg ah.
desired = ⌈serving × utilization / target⌉ (utilization saturates at 100 %); new instances serve after the boot delay[6]Filasho: tirada shaqaalaha, wakhtiyada adeegga, awoodda kaydka xogta, cabbirka kaashka, xawaaraha buuxinta, qiimaha iyo heerka khaladaadka dhismaha xun waa qiyamyo tusaale ah oo loogu talagalay adeeg shabakad cabbir dhexe ah.
Isku-xirayaal kale oo hawlgal oo moodelku isticmaalo.
16 workers/instance · app time 50 ms (+20 ms and +1 query with the feature on) · 2 queries/request · DB 4,000 queries/s · 20,000 hot objects · cache refill τ = 30 min (slower while the DB is saturated) · warm-up job +12 %/min for 6 min, +600 queries/s · timeout 2 s · 30 % low-priority traffic · bad build +5 % errors, ×1.25 CPU · rollback 5 min · $0.40 per instance-hour · scale-in by ≤ 20 % of the fleet after 10 quiet minutes · up to 40 instancesFilasho: tirada shaqaalaha, wakhtiyada adeegga, awoodda kaydka xogta, cabbirka kaashka, xawaaraha buuxinta, qiimaha iyo heerka khaladaadka dhismaha xun waa qiyamyo tusaale ah oo loogu talagalay adeeg shabakad cabbir dhexe ah.

Aan-kala-sooc: abuurahaha mulberry32 oo abuur leh; qaybinta la isticmaalay — isku-mid, tibaax (CDF rogan), caadi (Box–Muller), Poisson (Knuth). Abuurka waa la muujiyaa waana la wadaagi karaa.

Ilaha

  1. Site Reliability Engineering — Ch. 3 Embracing Risk (error budgets), Ch. 4 Service Level Objectives — Beyer, Jones, Petoff, Murphy (eds.), O'Reilly, 2016
  2. The Site Reliability Workbook — Ch. 5 Alerting on SLOs (burn rate; 14.4× over 1 h = 2 % of a 30-day budget) — Beyer, Murphy, Rensin, Kawahara, Thorne (eds.), O'Reilly, 2018
  3. Teletraffic Engineering Handbook — Erlang C formula; waiting-time distribution for M/M/n, FCFS — ITU-D Study Group 2 Question 16/2 (V. B. Iversen), 2005
  4. J. D. C. Little — A Proof for the Queuing Formula: L = λW — Operations Research 9(3):383–387, 1961
  5. J. Jung, A. W. Berger, H. Balakrishnan — Modeling TTL-based Internet Caches — IEEE INFOCOM 2003, 2003
  6. M. Harchol-Balter — Performance Modeling and Design of Computer Systems: Queueing Theory in Action (M/M/k, server farms, capacity provisioning) — Cambridge University Press, 2013

Yaa shaqadan ka shaqeeya

Moodel waxbarasho — looma isticmaalo go’aamo hawlgal. Goobaha dhabta ahi waxay joogto kasta u hagaajiyaan qalabkooda iyo xogtooda.