Lottery staff and vendors overloaded by sales data
By Kate Northrop
The Minnesota Lottery confirmed yesterday that the cause for the delayed $2 billion Powerball drawing was a technical communications error brought on by an overwhelming amount of sales data.
If you recall the Powerball fever that took over the nation this past week, you'll also remember the ten-hour delay before the drawing took place.
When the drawing did finally occur Tuesday morning, it turns out that just one ticket sold in California matched all six numbers drawn and won the entire jackpot.
Millions of people went to bed on Monday night without a clue as to what was causing the interruption, but behind the scenes, the Minnesota Lottery was experiencing a technical error that prevented the drawing from taking place.
"For several hours, we couldn't complete our essential verification processes because of a communications problem with an external vendor's computer system," the Lottery explained. "Normally, issues like this are resolved within an hour. However, this time it took much longer."
Prior to every Powerball drawing, each participating jurisdiction must complete a series of security checks to ensure their sales data is accurate. If one lottery is unable to do so, the drawing must be delayed.
This part of the process is important because it verifies that all tickets have an equal chance to win and that the drawing pool is correct. "Balancing" means that each and every data entry is transmitted between two systems and must perfectly match. Normally, the Minnesota Lottery said, this occurs throughout the day in close to real time, but a communications error with their vendors meant that they could not confirm that their system had been accurately balanced.
On the night of the draw, these two vendors were not able to communicate in real time, which meant there was a delay in balancing the data. The ten hour delay was due to both the telecommunications glitch and the time it took for the Lottery to process the voluminous amount of sales data once communications was reestablished.
The lottery employs multiple third party vendors to ensure that the lottery is conducted fairly and that every ticket has a fair chance of winning. One such vendor is a central gaming system company responsible for printing tickets and recording lottery transactions to verify a ticket is authentic, and the other is an independent internal control system vendor that is responsible for checking the work of the aforementioned central gaming system.
Maintaining contracts with separate entities such as these is one of the checks the Lottery has in place to prevent anyone from either of these parties, whether it be staff from the Lottery or one of the vendors, from compromising the integrity of the games.
From the minute the Minnesota Lottery sold its last Powerball ticket at 9:00 pm Central Time until about 1:45 am on Nov. 8, Lottery employees were attempting to fix the communications problem, but the internal control system vendor determined that it would not be able to process the data it had collected throughout the day, the Lottery said in a statement.
As a result, the Lottery decided to "start over processing the entire day's sales data" at around 2:15 am. The process took about five and a half hours, and finally the Minnesota Lottery was able to confirm its sales data and balance its systems at 7:50 am. Then the Multi-State Lottery Association (MUSL) was given the clear to conduct the drawing, which took place four minutes later at 7:54 am.
"We sincerely apologize for the delay," the Lottery said in a press release. "We were just as excited as you were to learn about a new billionaire. However, we had no choice but to delay the drawing. Any error in the drawing would have been worse than the actual delay."
"We thank you for your patience and understanding," the Lottery continued. "While we can't promise another delay won't occur, we can promise that the security and integrity of the Minnesota Lottery will remain our top priority."
Crazy to think a different person(s) would have won the 2 billion jackpot if Minn did not mess up.
... and you know this how? The different person thingie? Isn't that like saying " had l put my left leg first through my pants this morning instead of my right- l may not have been in this three car accident with me rear ending the car in front of me!
This is what happens when you have millions of people trying to bet their way to Financial Freedom!
Because the same numbers would not have been drawn the night before.
What about when the numbers were being drawn on Tuesday,, After the second ball popped up the ball lady stops to make a congratulatory announcement, clearly disrupting the flow of the drawing. Her action changed the outcome of the drawing result. If she didn't interrupt the draw the result would have been different .
* Accept the reality we live in. The die is cast so to speak.
Yep.
Maybe, you're saying different numbers would have come out on Monday night instead of Tuesday morning?
As a result, the Lottery decided to "start over processing the entire day's sales data" at around 2:15 am. The process took about five and a half hours
They "sold its last Powerball ticket at 9:00 pm" and had an hour to balance the data but someone made the call to start over and that took five and a half hours.
My question is how was the drawing effected by one state not "balancing their data" in time?
But it could be worse had someone in Minnesota won after the delay. And the conspiracy people would be lighting up social media..
"Because the same numbers would not have been drawn the night before."
The "butterfly effect"
I guess you could say had the drawing been a minute earlier or a minute later the results would be different too. But without the results from a 10:59 PM EST drawing, we'll never know for sure.
I am not buying this story one bit. I already showed why.
It was some b.s. in the game...
I agree, I don't think the Powerball drawing numbers would have been any different if the drawing was held on Monday at the correct time instead of Tuesday morning. The numbers that were drawn were meant to be.
The day the numbers were drawn doesn't matter. What's more important is the time between when the individual balls are realeased into the chamber. If after the second ball came up they halt the process so she can read a commercial, the result will be different!
I'm actually surprised they fixed it in 10 hours instead taking a full day. It must have been all hands on deck and grab a friend. I don't expect they'll make PB or MM a daily game any time soon, on account of this problem. What if a large state like CA or NY had a glitch and it required a full day?
.. and all these clowns that are railing against my comments are to afraid to say of themselves " l would have been the winner, not someone from CA."😛
rigged rigged rigged they get away with it too
Whatever the real problem was in Minnesota, the results were still drawn in Florida and the winning ticket was sold in California.
Was the drawing still held Monday night or did they just wait to release the results? The host said have a good night, that's odd if the drawing was delayed till 8am. I think the drawings are done at a certain time each day no matter what since they are just videos and not live.
One will never know for sure but those videos could be days or months old.
Buying lottery tickets is dog eat dog, letting a person go before you can make that person a winner, and not you.
And Vice Versa!
They let you go first, then buy the Winning Ticket!
It wasn't just the amount of data. It was a telecommunications error that caused the original problem. And then when communications was finally restored, they found they couldn't just resume processing — they needed to start the entire process from the beginning. That need to start over turned a couple hours of delay into 9 hours.
* Now which Clown downgraded my last salient point- raise your hand, identify yourself!
I'll leave "the drawing was rigged" to the conspiracy crowd because IMO the delay in the drawing had no effect on the outcome. Hopefully a better process will be designed so something like that doesn't happen the next time the jackpot is over $billion.
Wow, your post was rated lower than the new kid on the block with the weird ID.
.. am l being reasonable? l think so, all l want to know is who are these folks that see fit to downgrade a legitimate post of mine. These guys are suggesting that the winner of the PB Jackpot from CA* was given an " advantage" by the delay, and all l am asking is " How?" Get back in here and explain themselves. Is that to much to ask? l think not!
"It wasn't just the amount of data. ... they needed to start the entire process from the beginning. "
So basically it was the amount of data.
"an "overwhelming amount of sales data" doesn't explain a thing "
Of course it does. The amount of data and the speed at which they can process the data determine how long it takes, and selling more tickets and not having a problem has nothing to do with what happens when there is a problem. For this run MN sales ranged from a low of $178 thousand to a high of $9.6 million. Regardless of the particular problem there's no doubt some overhead that's independent of the amount of data, and what they had to do could have taken less than 30 minutes or a bit more than 10 hours depending on which drawing had the problem. That it happened when sales were unusually high easily explains why it would take quite a bit of time.
"your post was rated lower than the new kid on the block with the weird ID. "
The new kid hasn't had nearly as much time to attract stalkers who vote as much on author as on content.
"The numbers that were drawn were meant to be. "
Feel free to explain why that makes more sense than the idea that all sorts of factors can have an affect on what happens.
"These guys are suggesting that the winner of the PB Jackpot from CA* was given an " advantage" "
The butterfly effect posits that all sorts of very minor differences based on when the drawing takes place can have an effect on the results. That doesn't mean that any ticket had more or less chance than any other, but that a drawing delay will produce a different result.
No argument from me. That was horrendous.
Funny several weeks ago not one peep that I remember when the Powerball drawing machine broke down in beginning of the drawing which delayed it for one hour. What can you expect from a state that can't count votes (chads)--Floriduh.
A "telecommunications error" more than likely means that a piece of networking hardware failed and went down.
The lottery terminal at your local lottery retailer is just one piece of many pieces of communications hardware that's connected in a chain of other network communications hardware in between the lottery terminal and a datcenter where a mainframe or a powerful server resides. It's the mainframe/server that actually processes the data sent back and forth over the network. If any one device in the network chain goes down due to either a hardware or software problem, then there can be no communication between the lottery terminal and the mainframe.
Data is transmitted over network devices in a chain of "boxes". (hardware) And while most wouldn't think it possible, an extra large volume of data can literally choke the network and can cause one single piece of equipment in the network chain to go down. When telecommunications was restored (a failing box in the network was fixed) they realized they needed to start over from scratch. Those of us that worked in the data processing industry know that networking problems such as this are common and so is the solution to the problem. (Starting over from scratch) G5
A better process (a bullet proof network) can be designed. It's a network so full of redundant equipment that when one box in the network fails, it's automatically "failed over" (switched over) to it's back up box. (Which costs a lot of $$ for it to sit there doing nothing while waiting for the primary box to fail)
A network like that is so expensive that the cost of it makes it prohibitive to all but the Federal Government. For instance the fleet of Space Shuttles were chock full of redundant hardware. Some boxes were even "double redundant" meaning there were two back up boxes because whatever the primary box did was very critical to keeping astronauts alive while in orbit.
So yes, while it can be done, there aint any private enterprise businesses that are willing to pay for it. G5
I would agree that redundant communications are needed, but disagree as to cost. It is actually cheap these days to have redundant architecture. There are ways to (relatively) cheaply build redundancy into every single part of an architecture. The state lotteries are largely reliant upon a few large lottery vendors to do all the work for them; the in-house IT staffs are not the lotteries' strength. So if the vendors are not good at it, then the lotteries will not be either.
If they were using Checkpoint Software they could have done a Checkpoint Restart which allows an application that crashes or fails for any reason to continue from the Last Checkpoint without data loss when it is restarted.
In other words: Defecation Happens!
Hope this helps
https://www.lotterypost.com/thread/340427
From "several weeks ago",completed with a simple LP search...It appears there are multiple "peeps".
Florida Chad issue from 20+ years ago:
The Palm Beach County supervisor of elections, Theresa LePore, a Democrat at the time, approved the infamous and confusing butterfly ballot, blamed for many Gore votes mistakenly being cast for conservative independent candidate Pat Buchanan. The year after the recount she changed her party affiliation to independent, saying in the South Florida Sun Sentinel that her position should be nonpartisan. “I wanted to take away any appearance or perception that may have been out there that I was partisan in any way,” LePore said. “And I’m a little disappointed, quite frankly, in the way the Democrats have treated me.” LePore ended a 34-year career with Palm Beach County Supervisor of Elections office in 2005 and has run her own company, T. A. LePore Consultants, for nine years. Along with business consulting, she serves on various boards including the Hispanic Chamber of Commerce of Palm Beach County and the South Florida Fair. She’s been Executive Director of the Miss South Florida Fair/Miss Palm Beach County Scholarship Competition for a decade. LePore talked about her experiences after the butterfly ballot in a 2013 article in Boca magazine. She spoke of death threats, lost friendships and said she never drives the same route twice and avoids public places during busy times.
Thanks for the update.
Stuff happens when old, outdated technology is used. We had IBM cards with chads from the 1970's in High school. I think the class was called Automated Data Processing but not 100% sure. It was fun.
Thanks for the info Todd. It's been 13 years since I last worked in a datacenter and I'm sure things have changed dramatically.
Most the network gear installed on the raised floor in my old datacenter was Cisco Systems equipment. I can recall IBM's Network Architects saying things like "We can always use more routers and switches, but that Cisco Sales Rep needs to sharpen his pencil for that to happen." At that time I think Cisco was in a position to price their machines however they wanted. I'm sure competition from F5 Networks and others has had some bearing on how Cisco prices their equipment these days. G5
Going to take a break from powerball. That was just sloppy.
Me too. Don't know about others, but when the big one is finally one feel like a balloon has been deflated.
Don't know when I'll play again, something new to me.
Me three...haven't played any lotto since then , not even scratch offs .
you can say anything after the fact are we still supposed to believe that a "weather balloon" crashed in Roswell in '47? *asking for a friend*
I dunno, the whole thing made me suspicious and me and my friends feel it's rigged. Don't think we'll be playing it as much.