I have tried searching for ODBC and JDBC with Aurora or other amazon databases, but all of the result I get back are how to set up a connect to those databases using ODBC or JDBC, not how to set up a connection from there to another database.

Hi Scott - have you checked out www.tacticalconnect.com, it's a saved search based connector that can send data directly to S3 buckets.

Which RDBMS are you actually going to use on RDS?

@Josh_SatoriReporting most of the data we are pulling is full tables (all records, but not all fields) and they are not backed by any saved searches

@scottvonduhn we can pull all tables/records that are important via searches. It's quite the opposite with ODBC where everything is not available. For example, our data model fully replicates netsuite for financial and operational reporting down to the transaction level detail.

It may be worth a look. We don’t really use the data we extract for reporting. It’s not so much transactional data we need, it’s entity and billing information (customers, contacts, billing accounts, subscriptions, subscription lines, and usage) primarily for verifying that SuiteBilling is appropriately set up for all customers. If you have any info on how it works, or a demo, sales contact,. etc. I’d be glad to pass it along to my admin and we can go from there. Our biggest concern has always been speed. We push 25-50 thousand usage records daily, and retrieve 500 thousand to a million usage records for our daily one month look back and verification. We can retrieve that volume of data in about 5-15 minutes via ODBC currently. As long as our new process is in the same ballpark, I think we’ll be fine.

I'm by no means an expert but I you should be able to do this using AWS Glue https://aws.amazon.com/about-aws/whats-new/2019/11/aws-glue-now-enables-you-to-bring-your-own-jdbc-drivers-to-your-glue-spark-etl-jobs/

we've done a proof of concept using workato as middleware to push to S3 buckets. In workato you need to install the JDBC driver on an on-premise agent which could be annoying for some organisations (we don't mind as we need On-premise agents for accessing our proprietary systems anyway)

@michoel excellent suggestion, thank you! We are looking at using AWS Glue to replace SSIS anyway, so this would be a good solution. Ideally, we’d like to avoid using S3 buckets and retrieve the data directly into tables. There’s little benefit for us in having data go to buckets, it just adds an extra step in the process.

I've had a look at the documentation and it's not as simple as dropping in the NS JDBC drivers. You have to write a connector

I’m fine with writing the code for that. It will be a worthwhile investment.