Anyone else seeing downtime with test drive accounts this morning? I noticed many go offline around midnight ET last night.

All of our TSTDRV accounts went offline

@Matt Bernstein I've been told by NS support that they have deactivated all test drive accounts on the old infra and to request new accounts.

That’s what we are doing. We are confused because we didn’t see any notice of this

Yeah we didn't get anything either

same issue here. got a case open, but no response yet.

This is what support told me. We're escalating to everyone we can think of at Oracle,

@Chris Simon We've also been told that we can provide a list of test drive account ids to reactivate, but it's subject to "approval"

Same - we sent in our list of accounts at 8:42 AM ET. Radio silence since then 😬

we received the same response you did @Chris Simon. also escalating further.

We're escalating as well.

Same

This is what I got for my last TSTDRV demo instance extension -

Please be advised that once the account expires on the 30th of December, you will need to request a new account because we can no longer extend legacy accounts beyond 2023. You may also opt to request for the OCI version of this account now.

I can't comprehend how there wasn't more communication from Oracle on this. They just forgot about the SDN program.

Several balls were dropped, for sure. Unreal.

Brute force way to clean up all the tstdrv and non-sdn approved accounts?

This will be funny once I know we haven't lost thousands of hours of work.

I worked for a partner last year and recall having to provide a list to NetSuite of tstdrv accounts we were still using. I no longer work for the partner, but retained access to a handful of accounts I've had access to for > 10 years. But any that I used to have access to which were not on that list are now gone. Only 1 remains and it was moved to OCI last month.

We were required to pare down our SDN accounts as well. Unfortunately, even the accounts we requested to keep are gone/inaccessible.

I'm curious to see how NetSuite defines "reactivating" a dev or test drive account. Are they just going to spin up a new instance, or actually migrate the old one to OCI?

I'm hoping it's just a flag, and then they can deal with the migration path later on

What worries me the most about this incident is that I think it shows just how little regard NetSuite has for its developer community. They have never made it easy for us, but this goes way beyond that. I really wish that they'd take a page out of ServiceNow or SalesForce's developer relations books. Setup better developer programs. Setup a developer evangelist position. Make it easy to get developer accounts.

Agreed!

Maybe no one remembered that test drive accounts are used as release accounts for production bundles and SuiteApps?

Quiet here too, although I did hear through the grapevine that Guido is aware.

Yeah - same - the SDN team is definitely aware

This is a result of the Santa Clara data center going offline, impacting all demo accounts internal and external. Accounts that are prefixed with TD are on OCI. There was a process in place to move all demo accounts internally and externally to OCI. Was the processed followed, who knows at this point. From the sounds of things not all accounts were able to be moved prior to the data center going offline and accounts currently disabled should be moved at some point, hopefully soon!

Has anyone gotten any updates yet from support or the SDN folks in the past hour? Silent on my side, aside from our relationship manager deferring us to the SDN team

Nobody from Netgain has heard anything

Nothing here

Just got this response on our SDN support case: We have escalated your request internally and are attaching you case to Issue #750752 for temporarily re-activating your provided SDN TSTDRV accounts. SDN Team will also reach out to you by usual channels to discuss your options for transitioning demo program to Oracle Cloud Infrastructure (OCI). We will inform you once your accounts are processed and temporarily activated. Thank you.

We got 1 of our accounts turned on through a back channel connection.

@Chris Simon Learn anything in terms of timing for everyone else from the connection? Still in the dark.

Same here for the most part @Seth Duda, though I did get a tiny bit further. I did get a response back on my support ticket saying that our case has gotten attached to #750752 (the same one Craig mentioned) and that they're working to temporarily reactivate our TSTDRV accounts. Also, I received two automated demopurger emails for two of our TSTDRV accounts that got deactivated. Still no access to them.

demopurger 😮

"All accounts requested will be re-enabled as quickly as we can." lol

whatever you do, don't inactivate all accounts sorry, your mic is cutting out. repeat that? "inactivate...all accounts"

Oh to be the poor soul who made this mistake today 😅 I am not jealous of them

TRUE, FALSE, null checkbox maybe? =D

It must have involved some data loss with the deactivation - otherwise it makes no sense it's taking this long to roll this back (or have to be done on a case by base basis). Going on 15 hours now.

Finally heard back from support with the same message Craig got:

We have escalated your request internally and are attaching your case to Issue #750752 for temporarily re-activating your provided SDN TSTDRV accounts. 

SDN Team will also reach out to you by usual channels to discuss your options for transitioning demo program to Oracle Cloud Infrastructure (OCI). We will inform you once your accounts are processed and temporarily activated.

I find their repeated use of the word "temporary" interesting to say the least...

Received the same messaging and noted the repeated use of "temporary" blob eyeroll.

Got the same message as well

Appears we’ve got access back to our accounts now - can’t validate if there’s any data loss yet

Same here -- access has been restored on our side! Support has only restored access to TSTDRV accounts that I've sent along account IDs for specifically as I still have a few more that are missing. Highly recommend anyone else who is missing accounts to send a full list of account IDs over to NetSuite if you haven't already done so

For us, in the email that acknowledged the reactivation they stated that the accounts will be online "until EOD January 12th, 2024". FYI in case it helps anyone.

No word yet what the transition will look like before Jan 12 to move to OCI.

This is what I got from our SDN rep

Please go in and create a new support ticket asap for new native OCI accounts to be provisioned as replacements for the classic accounts. You can then use SDF or existing integration methods to migrate the accounts. Once this is complete the classic accounts will be disabled and OCI will be replacement.

Does this not work for anyone else? Some of our products are still SuiteBundles, I know you can "convert" a bundle to an SDF account customization, but as far as I know there's no way to then update customers with a SuiteApp going forward without massive issues. Not to mention the massive time commitment to take this endeavor on in uhh... checks notes ... 4 days.

Got the same email. Livid does not begin to describe.

What happens if you don't upgrade customers before the old account becomes inactive again? (on to the new copied bundle)

The SACC can be provisioned to another account by requesting a new SDN account and then submitting a support case including the old account ID, new account ID, and the SuiteApp name and publisher ID. I’ve done this once without issue. It goes without saying that the SACC feature must be enabled in the new account first.

Thinking maybe we need an "sdn" slack channel? This whole thing is looking like it could be an ongoing clusterf#&$.

anyone figure out the best approach to exporting existing bundles through SDF without individually converting each to a suiteapp, first?

I'm working on documenting that process now. For my needs, I think I can use the process on page 125 of SuiteBundler Guide (I'm looking at the 2021.2 version).

I wonder how many partners haven't noticed this change yet.

I've requested new OCI TSTDRV accounts and plan to test methods described above: • SuiteBundler Products ◦ As Craig suggested - copy bundle from the old release account to new release account and attempt to push an update to an account already in the install base from the new release account bundle ▪︎ Honestly a little skeptical this works, as Seth alluded to - what if the account installed on an unmanaged version and when they go to update the source account is non-existent? • SuiteApp Products ◦ As Zo suggested, create a support case to get the SuiteApp moved over - glad to hear this has been done before at least I've pushed back with our SDN rep on this migration being our responsibility. If they can migrate a production account to OCI seamlessly, why can't they do the same for their SDN partner accounts? I'm mostly worried about our release account, but our sales teams have put countless hours into making our demo environments - they just have to start from scratch? I'd encourage others to ask similar questions of SDN.

We have yet to get anything on our side from the SDN team, but emails have been sent.

Chris, I think it's a little more involved than that... see also "Deprecating a Bundle" in the SuiteBundler Guide. I believe you need to do these steps before pushing out the new bundle to your install base.

Ahh yeah this is a great find - thanks Craig!

Following the recent account access issue and subsequent re-provisioning, are others experiencing intermittent "unexpected errors" with some or all of their impacted accounts?

I've seen one account that had been re-activated do this

I had one account doing this and I submitted a support request. Despite not having any updates on that case, the issue appears to no longer be present in that account. However, it is impacting another account that was just fine moments ago.

I reached out to my SDN relationship manager and oddly, they still don't have a migration plan they are able to share with us. Perhaps they are rethinking their plan given how poorly the above that Chris shared may have been received by other SDN partners...? Also worth noting, our relationship manager said the 12th is "misinformation" and that official communications are coming soon.

First off thank you all for this thread. We requested new OCI/"TDXXXXX" accounts earlier this morning to try to get ahead of this. Our applications are in SuiteBundler and we had planned to copy our current TSTDRV release account bundles to a new TD dev environment, then copy that bundle to a new TD release environment and set that bundle to depreciate the old TSTDRV release bundle. Our thought was this would allow us to continue to use NetSuite's "Single Development Account Environment Method" deployment pipeline as described in their SuiteBundler guide. However we were just stopped by the 3 dev account environment limit! I understand that this limit is described in the SDN docs but how is this supposed to work?! With the "Single Development Account Environment Method" design alone you need separate Dev, Deployment, and QA/Test environments. Then ideally a permanent account for sales demos and training internal consulting teams - otherwise your demo accounts risk getting nuked every few months. That's 4 accounts for a single SuiteApp. We have 3 major applications that we'd like to keep separate code bases for. I'm starting to wonder if I'm jumping the gun by taking action based on this thread as opposed to waiting on my SDN mgr? (who has been mostly silent on the matter) We're just a bit shook by the lock out last Friday!

Can you point me to the "3 dev account environment limit"? Which SDN doc? I just requested a set of 5 accounts... DEV, QA-Trailing, QA-Leading, RELEASE, and DEMO and haven't had any pushback (yet).

It's called out on your SDN renewal invoice

@adamkost I've seen that also in our documents, but it doesn't actually seem like NetSuite keeps track of how many TSTDRV accounts each SDN partner has. The only thing I was told by the person at NetSuite who provisions these accounts is that we can only request 3 at once, so if you need 6 you'll need to make 2 separate requests

@Joanne, ah, I hope this is true! As a data point we've probably had 10 or so TSTDRV permanent accounts for 8ish years. I requested 2 new TD permanent accounts 3-4 weeks ago that were provisioned no problem. Then today I requested 8 more on 8 separate SuiteDemo cases so we could get ready for this shift - 1 was granted and 7 were denied citing "YOU HAVE MAXIMIZED ALL 3 DEVELOPER DEMO ACCOUNTS. YOU HAVE TO OPTION TO CANCEL THE EXISTING DEV DEMO ACCOUNT THEN REQUEST A NEW ONE.". Worrying, but I'll try again tomorrow. 😅

If these accounts are for product dev, make sure you're going through the SDN support system to provision the accounts. My understanding is SDN tstdrv accounts are not the same as demo tstdrv accounts.

@Matias Lugli FYI.

Still no news on our side - no response when requesting creating new accounts and no response from the SDN side either.

Last comm was midday yesterday that my request was forwarded to SDN. No accounts provisioned yet. No other comm from NS SDN. Beyond frustrating.

I’ve been told that all requested accounts are being restored and moved to OCI, and that NetSuite is not going to force partners to self-migrate, nor force them to quickly/suddenly get on SDF now.

David, do you have the exact communication as this is different than the comm I received? The email we received stated we had two options: Option 1: Request new accounts and self-migrate. Option 2: Send request to SDN to have them migrate accounts. Caveat is we would not have access to the accounts for 3 weeks. We're choosing Option 1 as we cannot afford 3 weeks of lost productivity. The problem is, I requested the new SDN accounts midday on Monday and they have not yet been provisioned. I did just receive an email from SDN PAM that the January 12th deadline is no longer in play. The old/current SDN accounts will not be deactivated again on Jan 12. Unfortunately, I don't know if that means it's extended for days or weeks. Assuming more comm is coming from SDN shortly. Big assumption on my part.

3 weeks!?

Yes my info is from Netsuite directly, people who have authority over this process. You should reach out again to sdn or sdn management and request direct answers to your question.

for anyone not on the 24.1 call currently, they addressed the topic and noted: • if you have account(s) impacted, contact support immediately ◦ note the account ids in the support case • for those who received the jan 12 deadline message, no more jan 12 deadline ◦ accounts re-enabled by support will be moved to OCI

I was told that once you have your accounts re-activated, no further action is needed by the partner

^ that was called out, too

I guess this must be the official communication from NS

"We have identified an issue that may have impacted SDN developer accounts. We're currently addressing a fix to this issue, but please double check that you have access to your SDN developer accounts.

If you identify any issues with your developer account, please create an SDN Support request. In the support request, include the account ID(s) of the affected account(s) (TSTDRVxxxx)."

Hi -- is anyone able to access the call recording from yesterday's 24.1 SDN webinar on the SDN portal? The only thing I can see under "WHAT'S NEW" is

BfN Updates for Release 24.1 Webinar Recording

, which appears to be an older recording from December

It normally takes them a little while to post it