Big Data Processing : Bring data from on-prem to cloud

[SumanPrasad]

Reading records in 100s of thousands or millions requires major programming effort so that you do not max out on memory or CPU. EASYProcess takes care of most of those complications by giving you built in features and parameters.
In this example, we will see how large amounts of data will be brought from your JDE/ERP to your application in the cloud.
You must do this in a batch job. Typical use of such a job is to bring say master data from JDE (Item master, address book) to the cloud.

Use database Inquire with DataSize set to 1000000 bytes(1 MB). This will limit the package size that will be sent to the cloud. You don't want to send 1 million records at once as that will break many size limits along the way.
Set save to DB = True. This will save data to a temporary database instead of keeping it in memory. You do not want to keep a million records in memory as it will throw out-of-memory exceptions.
Set NoOfRecords to -1. This will read all records from item master based on keys provided.
When you run this service, it will read item master and when it reaches 1MB of data, it will send it to the cloud to save in a temp table (Save DB = True) and then read the next 1MB of data. When this service finishes, the entire item master will be written to our temp table in the cloud.

Read Item Master.png (200.49 KiB) Viewed 5528 times

Read Item Master.png (200.49 KiB) Viewed 5528 times

Then Create a loop on this Inquire and then process each record by inserting into a local database. Loop is designed to automatically read the temp table if Save DB = True true on the target service it is looping.
So it is all happening behind the scenes for you and there is no additional programming you will need for it.

Loop On ItemMaster.png (84.87 KiB) Viewed 5528 times

Loop On ItemMaster.png (84.87 KiB) Viewed 5528 times