In case it's helpful to anyone, NetSuite has confi...
# generalr
rgoldstein
02/15/2023, 3:23 PMIn case it's helpful to anyone, NetSuite has confirmed that we are indeed missing 30 minutes of data from yesterday. NetSuite was down around 12:10pm EST, but any data generated after 11:40am is missing for now.
‼️ 3
rgoldstein
02/15/2023, 3:24 PMPer NetSuite:
Thanks for pointing this out. It’s clear that there will be a data reconciliation effort that we’ll need to help you with as a follow on from today’s issue.
As you know, your account was restored via out disaster recovery process but the restoration point was about 30 minutes prior to the outage we experienced. In this scenario, we will typically provide you with a report or list of the transactions that were created in that 30-minute window. Our engineering team is still actively working to restore service to our customers affected by the issue, but we have let them know about your request for this data and will schedule a call with you tomorrow to provide further updates.
m
MySuite
02/15/2023, 3:24 PMWow. This is crazy. Is Boston still down? It looks like it when you visit status.netsuite.com
r
rgoldstein
02/15/2023, 3:25 PMYes - it seems like some but not all customers have been migrated to another data center (Santa Clara in our case). Fortunately we were back up after around 4 hours, but many others on Twitter are reporting that they are still not back online.
rgoldstein
02/15/2023, 3:26 PMBeing offline was terrible, but having missing data is far far worse. Huge reconciliation this morning in our warehouses and trying to piece together missing orders.
s
Sophie White
02/15/2023, 3:35 PMOur production environment is still down and I'm worried about the data loss when we are up and running here soon (hopefully). However, I noticed that both of our sandbox accounts are working and have been at least the last 12 hours (not sure if they have been the whole time) and wondering if they were migrated to Santa Clara since the outage or if they've always been at another location. Anyone know how to see history on that or how i would be able to tell?
a
AMWhite
02/15/2023, 3:53 PMWe're back up now (Santa Clara rather than Boston), but were actually missing transactions from 10:30 on rather than 10:40. Not a huge difference, but we have a lot of transactions in that 10 minute window.
r
rgoldstein
02/15/2023, 3:54 PMOff that sucks @AMWhite. I'm sure you've done this, but we just built a saved search showing every system transaction from yesterday with time stamps so we could see down to the second when our most recent transaction was.
👍 1
a
Anthony Moran
02/15/2023, 4:08 PM@rgoldstein Did you use a Transaction numbering audit log Saved Search?
y
YT
02/15/2023, 4:08 PMStill down Boston Data center
p
Pan Zhang
02/15/2023, 4:15 PMAnyone seen this below
Pan Zhang
02/15/2023, 4:15 PMmessage has been deleted
p
Pong Bautista
02/15/2023, 4:15 PMYes, that's what I see
j
Justin B
02/15/2023, 4:15 PMYes, same as of late
p
Pong Bautista
02/15/2023, 4:15 PMin my production accounts
p
Pan Zhang
02/15/2023, 4:16 PMCame back, but production account not available
p
Pong Bautista
02/15/2023, 4:16 PMUpdate - We continue to work to re-establish full services in that location and are targeting to restore service by 8:30am PT / 11:30am ET.
r
rgoldstein
02/15/2023, 4:16 PM@Anthony Moran - no, just a mainline transaction search for all transactions created on 2/14 sorted by “date created”
✅ 1
s
Sophie White
02/15/2023, 4:31 PMyes, that's what we are seeing as well for our production account as of the last 25 minutes
s
Simon Whittle
02/15/2023, 4:41 PMMy 1 affected client just came back up and is still listed as in the Boston DC. They have no data loss as they have transactions through 12.16 EST so I'm wondering if the data loss only affects accounts that were relocated to another DC. It would fit with a restore but it looks like not all accounts affected by the outage have been relocated to another DC.
p
Pan Zhang
02/15/2023, 4:44 PMGood thought
j
Justin B
02/15/2023, 4:46 PMWe came back up still in Boston DC and there does not seem to be any missing data
a
AMWhite
02/15/2023, 4:46 PM@rgoldstein Yep, that's how we figured out we were missing that additional 10 minutes.
p
Pan Zhang
02/15/2023, 4:50 PMHow did you find out what data center you are currently in within netsuite?
j
Justin B
02/15/2023, 4:50 PMSetup > Company > Company Information
p
Pan Zhang
02/15/2023, 4:51 PMThanks Justin, saw it.
r
rgoldstein
02/15/2023, 4:53 PM@Simon Whittle - I suspect you are precisely correct. In our case I guess I'm happy we got restore sooner even at the expense of the 30 minute data gap, but it would have been nice to have the option (or for someone at NS to proactively inform us rather than letting us figure it out ourselves).
y
YT
02/15/2023, 5:20 PMit went up now
s
Sophie White
02/15/2023, 5:36 PM@Simon Whittle this checks out for us as well. we didn't go back online until 11:31am EST but when we did we were in the boston data center and seem to have no loss of data
3 Views