Essay
A Timeout Hidden in the Logs
日志里藏着的超时

At 11:47 p.m., I was still in the office corridor, the deep-gray suit I had worn for the morning review now creased at the elbows. My laptop was tucked against one hip when the notification arrived: the weekly summary job had failed again. The report showed forty-three timeouts in six minutes. In Mumbai, the office was nearly empty; the air-conditioning had shut down, and only the server-room fan
Two hundred milliseconds
Every attempt died at exactly two hundred milliseconds, which made the cause look tidy. It was not. The worker called the profile service, missed the cache, and spent its remaining budget rebuilding the same object. I widened the trace, copied one request into a local test, and watched the timer stop while the process waited on a database connection.
An exact number can be the least useful clue in a log.
My first patch increased the timeout to one second. It made the graph green for eleven minutes, which is exactly how a false repair behaves. I reopened the pool metrics and found twelve waiting connections behind one slow lookup. The code was not demanding more time; it was asking for the same answer twelve times. I added a short-lived cache around the lookup and kept the timeout change for the re
What the graph could not say
To reproduce it, I wrote a test with twelve uncached profiles. The first run failed; the second passed because the cache remained between tests. I shut the test process, cleared the cache, and ran the sequence again. This time the failure returned exactly where it belonged. A bug I had mistaken for bad configuration finally had a shape I could hand to the team.
By two in the morning, the queue was draining at the rate we expected. I stayed long enough to watch three complete cycles, then sent the patch and a note about the cold-cache path. Beside the keyboard sat a paper cone of vada pav, its potato crumb softened by the office heat. I ate half without tasting it, then found the cafeteria kettle and made masala chai.
A green graph after one patch is not yet a repaired service.
I took the local train home, where the windows reflected my face and the few people who had also been working late. The fix did not require a grand rewrite. It required a small cache change, a test that failed for the right reason, and the patience to distrust an improvement that arrived too neatly. At 3:10, I finally closed the laptop.
The next morning, the summary completed in four minutes. I printed the final timing beside yesterday's failed run, then did twenty minutes of yoga at home before opening my inbox. My shoulders were tight from the night, but the queue was quiet. Afterward, I bought a sandwich near the station and remembered, halfway through it, that I had eaten nothing but chai and half a vada pav.
中文译文
晚上十一点四十七分,我还站在办公室走廊里。上午评审时穿的那件深灰西装,手肘处已经压出褶皱。通知响起时,我正把笔记本电脑夹在髋侧:每周汇总任务又失败了。报告显示六分钟内出现了四十三次超时。孟买的办公室几乎空了,空调也停了,只剩服务器室的风扇还在转。
两百毫秒
每一次尝试都恰好死在两百毫秒上,看起来像是整齐的原因,其实并非如此。任务调用用户资料服务,没有命中缓存,于是用剩余预算反复重建同一个对象。我把追踪范围拉宽,复制一条请求到本地测试环境,看着计时器停下,而进程仍在等待数据库连接。
日志里越精确的数字,有时越不是线索。
第一版补丁把超时从两百毫秒调到一秒。图表随之变绿,并在十一分钟内没有再出错,这正是假修复的样子。我重新打开连接池指标,发现一次缓慢查询背后堆着十二条等待连接。代码并没有要求更多时间,它只是把同一个答案要了十二遍。我在查询外加了短时缓存,并把超时调整留给重试分支。
图表说不出的部分
为了复现问题,我写了一个包含十二个未缓存资料的测试。第一次运行失败;第二次通过,因为缓存还留在两次测试之间。我关掉测试进程,清空缓存,再跑一遍。这一次,故障准确回到它该出现的位置。一个我曾误判成配置问题的缺陷,终于有了可以交给团队的形状。
凌晨两点,队列按预期速度排空。我一直等到三个完整周期跑完,才提交补丁,并附上关于冷缓存路径的说明。键盘旁放着一个纸袋装的瓦达帕,办公室的热气让土豆馅碎屑变得发软。我没尝出味道就吃了一半,随后找到茶水间的电热壶,煮了一杯玛莎拉茶。
补丁之后图表变绿,不等于服务已经修好。
我搭乘当地列车回家,车窗映出我的脸,也映出少数同样晚归的人。这个修复不需要大改代码。它需要一处不大的缓存改动、一个会因为正确原因而失败的测试,以及耐心,去怀疑一份来得过于漂亮的改进。凌晨三点十分,我终于合上电脑。
第二天早上,汇总任务在四分钟内完成。我把最终耗时打印出来,和昨天失败的运行放在一起,然后在家做了二十分钟瑜伽,才打开收件箱。肩膀因熬夜仍然发紧,但队列很安静。运动结束后,我在车站附近买了一份三明治,吃到一半才想起,自己除了茶和半个瓦达帕,什么也没吃。




