Monitoring - We've identified the root cause of the issue and have mitigated it. We have also kicked off a backfill that will run over the weekend. We are shooting to have all events back in order and up to date by Monday morning. Expect updates over the weekend on the progress of the backfill of missing events. Thank you all for your patience and we hope you enjoy the rest of your Friday and the weekend!
Jul 19, 2024 - 22:53 UTC
Jul 19, 2024 - 22:53 UTC
Update - We continue investigating. We are close to understand the reason behind the event ingestion problem. It seems the root cause is not in the Kafka table engine, but on our write path to the distributed tables.
Events ingestion has been resumed, but it's going slowly to avoid those events disappearing, so there will be lag in the ingestion for some hours. We are working on pushing another patch to fix the lag.
After that is solved, we'll start the event backfill for the missing dates.
Jul 19, 2024 - 11:36 UTC
Events ingestion has been resumed, but it's going slowly to avoid those events disappearing, so there will be lag in the ingestion for some hours. We are working on pushing another patch to fix the lag.
After that is solved, we'll start the event backfill for the missing dates.
Jul 19, 2024 - 11:36 UTC
Update - We are investigating an issue with our kafka table engines and have purposely induced lag on our pipeline. All events are safe and will show up after this investigation, but for the moment we will fall behind on processing events and you will notice the last few hours missing in your reporting.
Jul 18, 2024 - 16:51 UTC
Jul 18, 2024 - 16:51 UTC
Update - We have started event recovery.
Data may be missing since 2024-07-17 at 21:00 UTC. The missing events will eventually be available for querying.
We are now working on pushing a fix to avoid this happening again.
Jul 18, 2024 - 13:29 UTC
Data may be missing since 2024-07-17 at 21:00 UTC. The missing events will eventually be available for querying.
We are now working on pushing a fix to avoid this happening again.
Jul 18, 2024 - 13:29 UTC
Investigating - We've spotted that the events ingested are lower than expected. We are identifying the root cause of the issue.
No data have been lost and we are already tracing a plan to recover it, identifying the impacted dates.
Jul 18, 2024 - 12:34 UTC
No data have been lost and we are already tracing a plan to recover it, identifying the impacted dates.
Jul 18, 2024 - 12:34 UTC
PostHog.com
?
Operational
US Cloud 🇺🇸
Operational
App
?
Operational
Event and Data Ingestion
Operational
Feature Flags and Experiments
?
Operational
EU Cloud 🇪🇺
Operational
App
?
Operational
Event and Data Ingestion
Operational
Feature Flags and Experiments
?
Operational
Support APIs
Operational
Update Service
Operational
License Server
Operational
AWS US 🇺🇸
Operational
AWS ec2-us-east-1
Operational
AWS elb-us-east-1
Operational
AWS rds-us-east-1
Operational
AWS elasticache-us-east-1
Operational
AWS kafka-us-east-1
Operational
AWS EU 🇪🇺
Operational
AWS elb-eu-central-1
Operational
AWS elasticache-eu-central-1
Operational
AWS rds-eu-central-1
Operational
AWS ec2-eu-central-1
Operational
AWS kafka-eu-central-1
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.