Evaluating D-SHAs with Write Connectors

Introduction

If you're using the S3 Write Connector in Ascend, it's important to understand how D-SHAs (a way of detecting changes in data) work and how they can affect your data processing. Currently, the S3 Write Connector doesn't evaluate D-SHAs when writing data to a destination system by default.

How to ensure D-SHAs are evaluated

To ensure that D-SHAs are evaluated before writing to the destination system, it's necessary to add a "no-op" Transform (which simply passes through partitions, without changing the data) before the write connector. This Transform will evaluate the D-SHAs and prevent further processing if they haven't changed. This ensures that you don't write unnecessary data to the destination system.

Handling the new 'upsert' feature

You'll need to perform two transformations (data with one partition -> partitioned data -> partitioned data for Data SHAs -> S3 write connector) to use D-SHAs before a write connector, due to the new 'upsert' feature of S3 Write Connector.

Conclusion

In summary, if you're using the S3 Write Connector in Ascend and want to use D-SHAs to detect changes in your data before writing to a destination system, you'll need to add a no-op transform before the Write Connector. This will ensure that D-SHAs are evaluated and that only necessary data is written to the destination system.