Loading... <p> <span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif"> </span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">为今年</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">5</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">月冠上多事之夏的名头已是无可厚非的一件事,自支付宝光纤被挖断后</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">携程又暴出全站瘫痪的风波</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">从</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">5/28 11:00</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">开始,直到晚上</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">11</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">:</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">29</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">分才全面恢复</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">.</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">互联网也是谣言四起</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">纷纷猜测百度腾讯谁会是下一个灾难的受害者。暂切抛开这些玩笑言论</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">就携程本次事情引发的思考太多</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">前车之鉴后事之师</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">如果携程的事情发生到我们身上</span><span style="text-indent: 24px;font-size: 12px;font-family: Cambria, serif">,</span><span style="text-indent: 24px;font-size: 12px;font-family: 微软雅黑, sans-serif">我们该怎么办,抱着这些问题我们来思考</span></p> <h1>1<span> </span><span style="font-family:宋体">全站瘫痪的原因分析</span></h1> <p style="text-indent: 36px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">自携程全站异常后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">整个</span><span style="font-size: 12px;font-family: Cambria, serif">IT</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">圈像炸了一样的在疯传着各样</span><span style="font-size: 12px;font-family: Cambria, serif">YY</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">流言</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">前离职高管报复行为</span><span style="font-size: 12px;font-family: Cambria, serif">;</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">公司高管睡了运维小哥老婆</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">运维小哥一怒之下物理格式化硬盘</span><span style="font-size: 12px;font-family: Cambria, serif">;</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程收购艺龙后引发商业对手黑客攻击入侵代码</span><span style="font-size: 12px;font-family: Cambria, serif">;</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">总之众说纷纭</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但简单分析即可看到所有的分析均是定位破坏性非常大的行为</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">因为自携程出事后</span><span style="font-size: 12px;font-family: Cambria, serif">3h</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">后依旧无法对外提供正常服务</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这点对外界的影响非常大</span><span style="font-size: 12px;font-family: Cambria, serif">,IT</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">互联网行为是以分钟计算金钱</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">以流量衡量价值的行业</span><span style="font-size: 12px;font-family: Cambria, serif">,3h</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">对于普通行业可能再正常不过</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但对互联网行业而言简直是一场灾难</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">况且携程旅游行业的出名度</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">引发巨大舆论也是再正常不过</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">大家出于破坏性大的角度考虑也是再正常不过的角度</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但最终官方给出的解释却让人大跌眼镜</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">运维人员误发布导致</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">好吧</span><span style="font-size: 12px;font-family: Cambria, serif">~</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">黑锅总是需要有人来背的</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">从如下几个方面来分析吧</span><span style="font-size: 12px;font-family: Cambria, serif">~</span></p> <h2>1.1<span> </span><span style="font-family:宋体">安全问题</span></h2> <h3><span style="font-size:14px;line-height:173%">1.1.1<span> </span></span><span style="font-size:14px;line-height:173%"> </span><span style="font-family:宋体">人员安全</span></h3> <p><span style="font-size: 12px;font-family: Cambria, serif"> </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这里的人员安全是指什么级别的人在操作</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">什么样级别的运维能同时拥有对全业务的权限</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程发展至今</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">在外界看来只是一个简单的入口页面</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">其实内嵌的页面和功能很有可能是成百上千的</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这样庞大的业务体系如果直接交给一个新手来发布或者初级工程师来操作</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那其责任更多的需要规究于领导责任</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">任重道远</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">当然</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">从任何的角度来看</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这样的可能性很小</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但如果是资深工程师的误操作</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那会是什么原因导致这样严重的问题的</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这样的问题在</span><span style="font-size: 12px;font-family: Cambria, serif">dev,test,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">准生产环境为什么没有发现呢</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">即使如上几个环境均没有发现</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那最后一道保障</span><span style="font-size: 12px;font-family: Cambria, serif">–</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">灰度环境为什么也没有发现呢</span><span style="font-size: 12px;font-family: Cambria, serif">?</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如此发问下来</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">最少有一点是可以确认的</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程没有灰度环境或者本次发布一定没有走灰度发布</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">难道本次发布只是一次小发布</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">没有固化流程</span><span style="font-size: 12px;font-family: Cambria, serif">?</span></p> <p><em><span style="font-size: 12px;font-family: Cambria, serif"> </span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">案例</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">:</span></em></p> <p><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow"> X</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">公司</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">业务人员变更</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">主业务维护人在新人没有完全上手的情况下</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">疏于协助</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">致使业务发布项遗漏</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">导致业务维护延期</span></em></p> <h3>1.1.2<span> </span><span style="font-family:宋体">固化流程</span></h3> <p style="text-indent: 24px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">我们权且认为携程本次的发布是一次小规模发布所以没有走灰度</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">既然操作变更比较小</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那是否有固化的操作列表供运维人员发布使用呢</span><span style="font-size: 12px;font-family: Cambria, serif">,rm –rf /*</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">虽说低级</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但不代表类似的错误没有</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">身边的例子不胜枚举</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">固化的流程和操作列表是一次发布的保护神</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">所以我个人认为</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">即使灰度环境已经验证过了</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但依然出现如上的问题</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那只有一个解释</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">没有固化的发布列表</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">运维人员纯手工敲打键盘所致</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如此看来的话</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">真的是运维小哥的问题了</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">案例</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">:</span></em></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">X</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">公司某业务发布频繁</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">且时常有凌晨发布状况发生</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">致使业务运维和其它各部门疲于应对发布类工作且发布压力较大</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">终有一日</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">业务运维发布过程中发布</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">checklist</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">遗漏</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">且为极为重要的操作步骤</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">导致业务数据回档发布延期且投诉居多</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">对公司业务声誉收入造成不可挽回损失</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">.</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">后采用自动化类工具固化后</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">此类从误操作从根本上杜绝</span></em></p> <h3>1.1.3<span> </span><span style="font-family:宋体">灰度问题</span></h3> <p style="text-indent: 36px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如果说运维是生产环境最后一道保障</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那灰度环境就是生产线最后一道屏障</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">纯手工误操作和无灰度发布发布问题</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">不管从哪方面讲对运维都不是一个好的消息</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如果说纯手工误操作那最多说明携程招人甚</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但如果是灰度问题</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">那携程的整个业务体系一定是不专业的</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">浮躁的</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">在</span><span style="font-size: 12px;font-family: Cambria, serif">IT</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">快速发展的国内</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">整个互联网都以结果为导向</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">快速生产利益</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">快速上市圈钱</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">虽说互联网的体系在不断完善</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但大体系中一定有轻有重</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">利益当道</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">利益部门的地位和权重一定优于其它部门</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p style="text-indent: 36px"><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">案例</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">:</span></em></p> <p style="text-indent: 36px"><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">X</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">公司某业务发展迅速</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">业务急速扩张</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">各部门精力及资源严重偏向收益类功能开发和新用户推广类工作</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">疏于业务夯实业务基础类功能开发及现有功能优化和现有业务思考</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">全业务在规模非常大的情况下依然采用全服发布方式</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">多次出现</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">bug</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">但无</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">GM</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">类工具挽救</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">每次均采用回滚或延期发布方式解决问题</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">.</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">后采用灰度发布后影响面和损失缩小不足原来</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">1%</span></em></p> <h3>1.1.4<span> </span><span style="font-family:宋体">权限安全</span></h3> <p style="text-indent: 24px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">从官方后来的解释来看</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">没有因为软硬件系统后门或安全类原因导致</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但内部人员的安全究竟如何我们很难一语评价</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">早在刚到现任公司的时候</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">做的第一件事就是</span><span style="font-size: 12px;font-family: Cambria, serif">D/O</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">分离</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">多次的线上文件丢失</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">数据异常</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">均因为开发拥有对线上权限过大导致所有的问题以说不清</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">找不到结束</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">此后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">项目权限最小化及测试服环境权限最小化</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">业务虚拟化推广</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">此后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">妖魔鬼怪类的问题也离我们远去</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">再无踪影</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程数千人规模是否有类同问题还需要携程内部自查了</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">案例</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">:</span></em></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">X</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">公司某业务发布及程序运行用户均为</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">root,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">业务运维同学某次操作过程中因权限过大人误删除业务数据</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">所幸没有造成灾难性后果</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">这么看下来权限过大的问题是埋在地下的地雷</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">终究会有爆炸的那天</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,root</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">权限回收</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,D/O</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">分离</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">权限最小化</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">—</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">业务的保护伞</span></em></p> <p style="text-indent: 24px"><span style="font-size: 12px;font-family: Cambria, serif"> </span></p> <h2>1.2<span> </span><span style="font-family:宋体">业务融入</span></h2> <p style="text-indent: 24px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">据析携程业务部门和运维部门完全分开</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">代码的管理也只是部分代码交于运维</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">开发部能自己维护的代码一定是自己维护好</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">导致这种现象最大的问题在于利益切分</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">业务部门对服务部门依赖越大业务的利润就要分割的越多</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">想必公司层面最原始的想法是好的</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">希望从制度上彻底杜绝资源浪费的情况</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">各部门自力更生</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但导致的直接问题除了各部门资源重复外还有这次的问题应该是所有人难以相像的</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">业务最好一道保障伞在对业务不了解的情况下</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">在问题出现后无法快速调动响应</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">案例</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">:</span></em></p> <p style="text-indent: 24px"><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">某业务被赋给公司极高期望</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">各部门及领导均极为关注</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">压力随之分散之业务及各部门</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">一时之间造成利益部门极为强势的局面</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">服务部门均抱极为尴尬应对场面</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">终因一次不必要事故造成部门间冲突升级并造成极大影响力</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">各部门高级总监聚集召开紧急会议</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">甚至部分高层连夜飞回会议中心</span></em><em><span style="font-size: 12px;font-family: Cambria, serif;background: yellow">,</span></em><em><span style="font-size: 12px;font-family: 微软雅黑, sans-serif;background: yellow">终双方抛开前嫌放下姿态共同支持业务</span></em></p> <h2>1.3<span> </span><span style="font-family:宋体">回滚时间长</span></h2> <h3>1.3.1<span> </span><span style="font-family:宋体">备份问题</span></h3> <p style="text-indent: 24px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">代码拉取</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">更新</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">备份</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">发布</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">校验</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">是一个业务完整的发布流程</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">看似简单但技巧良多</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">其中的业务备份是运维和业务的保护伞</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如若携程本次问题真是</span><span style="font-size: 12px;font-family: Cambria, serif">rm –rf /</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">类似灾难性的操作</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">本机的备份是一定没有了</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但远程机器的备份难倒也丢失了吗</span><span style="font-size: 12px;font-family: Cambria, serif">?~</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">这实在有些让人费解</span><span style="font-size: 12px;font-family: Cambria, serif">. </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">镜像或者快照功能也是难得也是做本地的吗</span><span style="font-size: 12px;font-family: Cambria, serif">?</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">对于携程究竟何种技术细心导致本次细节问题至今仍无消息</span><span style="font-size: 12px;font-family: Cambria, serif">~</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">倒是</span><span style="font-size: 12px;font-family: Cambria, serif">UC-</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">王总和阿里</span><span style="font-size: 12px;font-family: Cambria, serif">–</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">智锦的分析实在合理到位且颇具前瞻性。</span></p> <h3>1.3.2<span> </span>SOA<span style="font-family:宋体">业务架构形态</span></h3> <p style="text-indent: 36px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">阿里资深运维工程师</span> <span style="font-size: 12px;font-family: 微软雅黑, sans-serif">智锦</span> <span style="font-size: 12px;font-family: 微软雅黑, sans-serif">早在携程出官网结论前已神预测到</span><span style="font-size: 12px;font-family: Cambria, serif">SOA</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">的体系架构是导致携程灾难复建慢的原因</span><span style="font-size: 12px;font-family: Cambria, serif">.</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">从业务最初形态到至今</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">早已不是单接口单应用</span><span style="font-size: 12px;font-family: Cambria, serif">AllInOne</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">的架构</span><span style="font-size: 12px;font-family: Cambria, serif">,SOA</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">带来业务隔离性的同时也带来了业务的灵活性复杂性</span><span style="font-size: 12px;font-family: Cambria, serif">, </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如</span><span style="font-size: 12px;font-family: Cambria, serif">UC</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">王总所说</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">随着业务的越来越大</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">人员迭代</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">业务的运维对业务只能增加不敢减少的也是本次恢复慢的主要原因</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">本次业务代码全丢失后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">本次的问题相当于从零开始</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">而且是在无数旧人迭代和官网瘫痪的巨大压力下完全重构业务</span><span style="font-size: 12px;font-family: Cambria, serif">. SOA</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">体系带来方便的同时也带来业务的复杂程度提高</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">任何武器都是双刃剑</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">使的招数则完全看掌握者</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">既然已经复杂了</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但全量备份</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">异地或者跨主机备份呢?无论从哪种角度解释都有很多招数可以迅速恢复</span><span style="font-size: 12px;font-family: Cambria, serif">~</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">到此侧面不同程度反应出携程的虚拟化和备份一定有不足的地方</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">如此两点够携程运维同学吃好大一锅了</span><span style="font-size: 12px;font-family: Cambria, serif">~</span></p> <p><span style="font-size: 12px;font-family: Cambria, serif"> </span></p> <h1>2<span> </span><span style="font-family:宋体">公关问题</span></h1> <p style="text-indent: 30px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">相对携程来说</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">阿里的公关不得不说是专业的</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">在阿里光纤被挖断分钟级别已经对外公布问题</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但携程则完全不一样</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">时至今日也只有对外一个简单的说法</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">宕机后数小时后期间公司公关部门完全无所作为</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">甚至连阿里挑衅的微博引导流量也完全哑炮状态</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">倒有一块看热闹的怀疑</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">不禁让人怀疑这公关部门是亲生的吗</span><span style="font-size: 12px;font-family: Cambria, serif">?</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">实在让人不以为然</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <h1>3<span> </span><span style="font-family:宋体">学习处</span></h1> <h2>3.1<span> </span><span style="font-family:宋体">静态有损服务</span></h2> <p style="text-indent: 30px"><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">好了</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">现在可以来谈谈携程本次问题优秀的地方</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">在业务完全宕机后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">作为一家大公司</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程的静态有损服务总算给自己留下最后一点尊严</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p><span style="font-size: 12px;font-family: Cambria, serif"> </span></p> <h2>3.2<span> </span><span style="font-family:宋体">请求导流到艺龙</span></h2> <p><span style="font-size: 12px;font-family: Cambria, serif"> </span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">请求导流</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程对艺龙的收购难道是对未来的预知吗</span><span style="font-size: 12px;font-family: Cambria, serif">?</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">开句玩笑话了</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">携程在网站全瘫痪后</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">迅速把流量导向自己刚收购的艺龙</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">虽然艺龙不能承受随之而来的并发</span><span style="font-size: 12px;font-family: Cambria, serif">,</span><span style="font-size: 12px;font-family: 微软雅黑, sans-serif">但思路最少是值得参考的</span><span style="font-size: 12px;font-family: Cambria, serif">.</span></p> <p></p> <p></p> 最后修改:2021 年 12 月 10 日 10 : 53 AM © 允许规范转载 赞赏 如果觉得我的文章对你有用,请随意赞赏 赞赏作者 支付宝微信