30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Hi, I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.can you let know anything wrong here?
I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates
We are logging pipeline errors /orchestrated task errors in database . Now often, we see error messages like - “for actual error , open the logs “ as runtime errors.Per our solution , this gets logged into the database . We don’t want that ..Instead we want the actual errors to get logged . How can we do that ?
"Can values be inserted into another database table from a temporary table that is created inside a function?"
We are excited to announce our new 30-day trial of the full StreamSets Platform! 🎉 Anyone who is interested in trying out our cloud-native platform for data integration can sign up for the trial - no restrictions. If you’re interested in sharing this news with a colleague or want to learn more, check out our latest blog post.
Since SDC 5.4.0 was released, there is available a new origin stage Oracle CDC which will replace the old stage Oracle CDC Client. You can have a look into the release notes where it is explained in more detail.StreamSets recommends using the new Oracle CDC origin for all new development. So if you would like to migrate to the new Oracle CDC and take benefit of its improvements, you should take into account two new drivers needed for the new stage: orai18n.jar and orai18n-mapping.jar, so please, check our documentation to install the required Libraries.Otherwise you can get errors running your pipeline as the following:Pipeline initialization error: java.lang.NoClassDefFoundError: oracle/i18n/util/LocaleMapperAnd remember to restart SDC after installing the new drivers.You can always check our documentation for further information.
I have a simple pipeline from oracle multi table into snowflake destination. This error only happens when the destination table is not present but it does create the table with PK. After pipeline ran and I can truncate the table and re-run the pipeline without any issues. Any ideas?
Hello!I'm currently trying out the free (open source version) of SDC. When I created an empty pipeline and added a cron scheduler and JavaScript Evaluator and tried to run it, I encountered the following error:Pipeline Status: STARTING_ERROR: java.lang.NoClassDefFoundError: Could not initialize class com.streamsets.pipeline.stage.processor.javascript.Java8JavaScriptObjectFactoryI was under the impression that using the JavaScript Evaluator did not require any additional package installations. Is there an additional step that needs to be taken?I need some help. Thank you.
Hello,I'm currently using the HTTP Client Processor in StreamSets Data Collector and I've encountered an issue with the retry mechanism. Despite configuring the processor to retry on receiving certain HTTP status codes, it doesn't seem to be doing so.Specifically, I've set the processor to retry immediately when it receives a 409 status code, with a maximum of 2 retries. However, when the processor receives a 409 status code, it doesn't retry the request and instead gives the following error: "HTTP_101: Applying passthrough and error policy on status configuration".I've checked the pipeline configuration. I'm not sure why the processor isn't retrying as expected.Has anyone else encountered this issue? Any insights or suggestions would be greatly appreciated.Thank you in advance for your help.
Hi Team While installing DC Tarball engine using install script from Dataops platform, it asks for download and install directory at run time.Is there a way we could avoid passing it during execution and pass it in install script or use current directory?
Error: Source : hello everyone, I want to pull data from the source to the warehouse, in the source there is a "customer_id" column, when the streamsets are run why is there an invalid customer error? in streamsets I also flag based on the "id" column. please help, thank you
Hi Team,We have some pipelines, which has some configuration like below, How can I get the exact plain test value for these parameter property value? Also how and where can I set these kind of values?
I read from an SFTP server and when writing to S3 my files get renamed. How can I retain the original file name?
Become a leader!
Already have an account? Login
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.