Anyone remember "Prime Day" this year?

[Rob]

And what a complete disaster it turned out to be for Amazon and the customer experience?

Well....the report is in and even the big players in this game cannot seem to make a better wheel so to speak. I keep hearing people and companies pitching their reasons to go to the cloud and get out of these crazy contracts with vendors like Oracle, IBM, Microsoft, etc....but these lessons come at a cost and you can bet Amazon won't make that mistake again.

Amazon is learning how hard it can be to move off of Oracle's database software.

On Prime Day, while the e-retailer was dealing with a major website glitch that slowed sales, the company was also dealing with a technical problem in Ohio at one of its biggest warehouses, leading to thousands of delayed package deliveries, according to an internal report obtained by CNBC.

The problem was in large part due to Amazon's migration from Oracle's database to its own technology, the documents show. The outage underscores the challenge Amazon faces as it looks to move completely off Oracle's database by 2020, and how difficult it is to re-create that level of reliability. It also shows that Oracle's database is more efficient in some aspects than Amazon's rival software, a point that Oracle will likely emphasize during this week's annual OpenWorld conference in San Francisco.

Following the Prime Day outage, Amazon engineers filled out a 25-page report, which Amazon calls a correction of error. It's a standard process that Amazon uses to try to understand why a major incident took place and how to keep it from happening in the future.

The report shows that Amazon struggled to identify the root cause of the Prime Day issue because of a feature it lost after the database was moved over. It also failed to come up with a contingency plan in case of an error in its newly installed database, called Aurora PostgreSQL, the documents show.

In one question, engineers were asked why Amazon's warehouse database didn't face the same problem "during the previous peak when it was on Oracle." They responded by saying that "Oracle and Aurora PostgreSQL are two different [database] technologies" that handle "savepoints" differently.

Savepoints are an important database tool for tracking and recovering individual transactions. On Prime Day, an excessive number of savepoints was created, and Amazon's Aurora software wasn't able to handle the pressure, slowing down the overall database performance, the report said.

Could have happened anyway

"It's quite possible the outage would not have occurred if Amazon had stuck with Oracle," said Matt Caesar, a computer science professor at the University of Illinois at Urbana-Champaign, after CNBC shared the details of the document. "Also, it appears they would have been able to diagnose the problem sooner if they were using Oracle's database, which could possibly have reduced the outage duration."

An Amazon spokesperson played down the issue in an emailed statement and said there was no outage, even though the internal document states that the database "degradation resulted in lags and complete outages."

"It is important to point out that there was never an outage at the facility, and the issue only resulted in delaying shipping of about one percent of packages for a short period of time," the spokesperson said. "This issue was quickly diagnosed and resolved."

The Ohio warehouse is the largest of the 13 warehouses that moved its database off Oracle prior to Prime Day. During the Prime Day period, it handled over 1.1 million packages per day, the documents say. All services and software that handle inventory and shipping data had been migrated to Aurora in those warehouses.

The outage, which lasted for hours on Prime Day, resulted in over 15,000 delayed packages and roughly $90,000 in wasted labor costs, according to the report. Those costs don't include all the lost hours spent by engineers troubleshooting and fixing the errors or any potential lost sales.

In a section titled, "Lessons Learned," Amazon engineers wrote that, "Savepoint behaves differently in Aurora PostgreSQL than in Oracle," suggesting Oracle's software would have handled the issue more efficiently. It also says SQL statement data did not exist for analysis in PostgreSQL, and having access to that data "would have helped pinpoint" the root cause of the problem.

The outage may have been less severe had Amazon been more prepared. In one part of the document, the company said it "took a long time to mitigate" the problem because of a "lack of a reaction plan when the underlying PostgreSQL DB experiences performance issues." The document also said a "well-established reaction plan or runbook" could have helped "mitigate the impact sooner."

"My guess is that they changed databases a while ago, didn't test the exact load model that occurred during their Amazon Prime Day and got surprised, badly," Henning Schulzrinne, a computer science professor at Columbia University, said after reviewing the document.

Amazon and Oracle have been in a heated battle of words in recent years, as Amazon has expanded its software offerings to more directly compete with Oracle. CNBC reported in August that Amazon is working on moving its entire database off Oracle by early 2020.

'It's really, really hard'

Oracle Chairman and co-founder Larry Ellison isn't buying it. On the company's earnings call in December, Ellison said Amazon "is not moving off of Oracle." He reiterated his point at an August event, saying, "I don't think they can do it."

"They've had 10 years to get off Oracle, and they're still on Oracle," he said. "And it's not going to be easy for them to use their own technology. It's not going to be cost-effective. I mean, it's really, really hard."

Patrick Moorhead, principal analyst at Moor Insights & Strategy, said the incident shows how hard it is for older applications, like those used in Amazon's warehouses, to move off Oracle, which has spent decades working with the world's largest enterprises.

"AWS Aurora is designed for forward-looking applications and Oracle for more legacy applications," he said.

[Mike Mercury]

[Thunder22]

What people are missing here is that you can't take legacy applications and swap out legacy components with modern components without re-writing the application. Monolithic on-prem/non-cloud applications are designed and engineered very very differently than modern "cloud applications". If AWS didn't modernize the warehouse application and only swapped out the back end database, and then failed to properly stress test it, that's not an indictment of cloud based applications, that's an indictment of AWS' poor sw architecture/engineering methods.

[DAB]

Yeah, what ^^^ said.

I don’t understand it, but I’m sure he’s right.

[Thunder22]

Originally Posted by DAB

Yeah, what ^^^ said.

I don’t understand it, but I’m sure he’s right.

Translation: aws sw engineers made a mistake, not the aws platform

[Uncle Meat]

The military is moving more and more of their apps to the cloud. Immediately there's a drastic performance hit when they do this. Moved our Outlook to the cloud and it hasn't been the same since. The cloud sucks, don't buy into the hype.

U.M.

[Cybercowboy]

Thing is, Oracle is losing customers like AWS simply because they charge outrageous amounts of money for licensing. Insane amounts. The AWS engineers will get their replacement database fixed and eventually it will run rings around Oracle's database solution simply because it will be optimized for how AWS uses it. Oracle could simply offer licenses that make trying to do it in-house not even worth considering. But no, they are never going to do that.

20+ years ago, it's pretty much how my little company I worked for then basically put several big players out of the medical dictation industry. Dictaphone, Lanier, Sudbury Systems - their dictation systems were expense as f**k. And they really didn't do anything special. We came in with a PC-based solution that could scale up as needed. It was as reliable as their stuff and we could charge about 1/5th the cost and still make wheelbarrows full of money.

[Thunder22]

Oracle's licensing algorithm has been insane since day 1 and changes frequently. I'm not a fan at all.

[Rob]

Being an Oracle DBA. I agree about the pricing, we get a 96% discount on our annual fee and still pay several hundred thousand dollars a year in fees.

[DAB]

Larry needs the money for his mansion and his yacht (America's Cup). get to work you minions!

[yell01]

.