mail@pastecode.io avatar
6 months ago
2.8 kB
Open AI Incident
OpenAI confirmed that they faced a bug that caused ChatGPT to expose payment details for paid users, in addition to leaking conversation histories from random users.
OpenAI said: “The bug was discovered in the Redis client open-source library,
Because this bug leads to the leakage of customer data, it is classified as a vulnerability and it is indexed with these CVEs:

What is Redis?

Redis is an open-source, in-memory, key-value data store most commonly used as a primary database, cache, message broker, and queue. Redis delivers sub-millisecond response times, enabling fast and powerful real-time applications.

OpenAI uses Redis to cache user information across its servers so it doesn’t need to query its database for every request. OpenAI further uses Redis Cluster to fairly distribute load across multiple Redis instances. 
but still, the bug is not with the Redis itself, the bug is within redis.py, which is a library that used to interact with the Redis database.

An example that shows the difference between Redis itself and redis.py:

OpenAI describes the bug as follows:
We use the redis.py library to interface with Redis from our Python server, which runs with Asyncio. The library maintains a shared pool of connections between the server and the cluster and recycles a connection to be used for another request once done.
When using Asyncio, requests and responses with redis-py behave as two queues: the caller pushes a request onto the incoming queue and will pop a response from the outgoing queue, and then return the connection to the pool.
If a request is canceled after the request is pushed onto the incoming queue, but before the response popped from the outgoing queue, we see our bug: the connection thus becomes corrupted and the next response that’s dequeued for an unrelated request can receive data left behind in the connection. 
In most cases, this results in an unrecoverable server error, and the user will have to try their request again. 
But in some cases, the corrupted data happens to match the data type the requester was expecting, and so what gets returned from the cache appears valid, even if it belongs to another user.
At 1 a.m. Pacific time on Monday, March 20, we inadvertently introduced a change to our server that caused a spike in Redis request cancellations. This created a small probability for each connection to return bad data.

My recommendation:

As a security architect if I was responsible to review a system that uses redis.py my recommendation will be to not use package older than 4.5.3 

March 20 ChatGPT outage: Here’s what happened (openai.com)
NVD - CVE-2023-28858 (nist.gov)
NVD - CVE-2023-28859 (nist.gov)