Two clocks for one GitLab.com loss: 00:00 UTC and 23:30 UTC, at least 5,000 projects in the postmortem, 5,037 in the log

The named disagreement is the loss account, not a second outage. The 10 February 2017 postmortem’s introduction says modifications to projects, comments, user accounts, issues, and snippets between 17:20 and 00:00 UTC on 31 January were lost, and that the best estimate was roughly 5,000 projects, 5,000 comments, and 700 new user accounts. The data-loss section of the same postmortem says projects, issues, snippets, and similar rows created between 17:20 UTC and 23:30 UTC were lost, and that the exact loss is hard to estimate: at least 5,000 projects, 5,000 comments, and roughly 700 users. The incident log’s impact list prints ±6 hours, 5,037 projects (4,613 regular, 74 forks, and 350 imports, labeled roughly), ±4,979 comments (so ±5,000), and 707 users potentially. The Register quotes those log figures the same day, including about 4,979 comments and 707 users that were hard to tell from the Kibana logs, and its standfirst compresses that to about six hours for ~707 users. “700 new user accounts,” “roughly 700 users,” and “707 users lost potentially” are three wordings. The Register also reports a PR line, less than 1% of the user base, and says PR described the loss as peripheral metadata written during a 6-hour window. The opened postmortem does not print that percent or the word peripheral. Its nouns are the database objects above.

The fault on the primary is removal of the PostgreSQL data directory on the production host, meant for the secondary. The postmortem times that at about 23:30 UTC and says the engineer stopped the process a second or two after noticing, by which point around 300 GB had already been removed. The log times the termination at 23:27 UTC and says of around 310 GB only about 4.5 GB was left. The signal that catches the mistake is the operator seeing the host. It misses every object already unlinked: neither document lists the relations inside those gigabytes, and neither equates the gigabytes with the project count. Earlier that night, strace on pg_basebackup showed a hang in poll and, the postmortem says, no other meaningful information that might have explained why. That trace does not name the later removal, and it does not say why the client waited. The postmortem says the hang was normal behaviour that was not clearly documented: pg_basebackup waits silently for the primary to send replication data. The trace still did not show that on its own.

The backup signals that should have shown an empty dump did not produce a restorable backup. The postmortem says the daily S3 bucket was empty because the procedure ran pg_dump 9.2 against PostgreSQL 9.6, which it treats as a major-version mismatch, on an application server where Omnibus saw no PG_VERSION and defaulted to 9.2. Cron mail was the signal for that failure. It missed the failure because DMARC was not enabled for those messages and the receiver rejected them. Azure disk snapshots were on for the NFS servers and not for any database server; the postmortem says other backup procedures were assumed to be enough. Replication could not be a failover copy either: the secondary’s data directory had been wiped to rebuild it, and WAL segments the secondary needed were already gone because GitLab.com was not using WAL archiving. What remained was the LVM snapshot taken at about 17:20 UTC and loaded into staging. Copying that staging data to production took around 18 hours. The postmortem says those Azure classic disks were throttled to around 60 Mbps, that there was no network or processor bottleneck, and that the bottleneck was the drives. The log line dated 2017/02/01 04:00, which does not print a timezone, says the transfer was slowed by two factors, network I/O between us-east and us-east-2 and a disk throughput cap of 60 Mb/s. Neither sentence is a measured byte rate. The postmortem says the database was restored without webhooks at 17:00 UTC on 1 February and that final procedures, including webhooks, finished around 18:00 UTC. The log prints 2017/02/02 16:28 for transfer complete, on a capture whose URL is timestamped 2017-02-02 03:47:19 UTC. A separate log estimate, ±20 hours, refers to an initial pg_basebackup sync while staging was being set up, not to this production copy.

What the log can prove is the text it printed, not which rows stayed gone. The user figure is a Kibana estimate: 707 users lost potentially, hard to tell for certain. The project total is explicitly rough, and the same sentence says Git repositories were not lost, so projects whose user or group already existed could be recreated without their issues. In that capture, creating those project records is still an open checklist item. By the 10 February postmortem, a procedure had restored projects from Git repositories still on the NFS cluster, only in basic form, without issues or merge requests, and the exact loss was still described as hard to estimate. The 5,000 and 5,037 figures are therefore not a count of objects that remained absent. No issue count, snippet count, merge-request count, or webhook count is printed. The log says webhooks created before 17:20 on 31 January were restored and later ones were lost. The Register prints that cutoff as 5.20pm with no timezone in the bullet. The postmortem’s recovery steps say the process would increment sequences by 100,000. The log checks off an increment of 10k. This page does not decide which increment ran, and it does not treat either document as a census.