UpdateStatus/WriteHistory Improvements
Posted: December 19th, 2022, 5:28 pm
I was talking with Cameron recently and realized the UpdateStatus is deprecated and replaced with WriteHistory which has way less inputs. The WriteHistory service should receive some improvements, but it depends how we plan to use it going forward. I saw 2 possible options and outlined them below
Update Status:
Write Status:
Option 1: WriteHistory is a basic use case service
If its just the one you use for basic case and the UpdateStatus weren't deprecated, I'd be okay with that. Though there are still some changes it should have.
It needs ErrorCount to be added. Because we should never promote for the developer to not do their error handling
It doesnt need the option to reset messages and counts because those are advanced use cases. throwing out the data youve collected so far is a little dangerous and shouldnt be encouraged unless the developer knows what they're doing
So Write status would have the following inputs
In this case, its worth looking at the existing inputs for the UpdateStatus and how those are used today:
Name: can be removed
This can be removed if EASYProcess knows which batch job is running. We don't have any complex need to update another batch job from the one that is running.
EndTime/JobEnded: Worth discussing
I think that the end time/JobEnded flag doesn't do anything. I never tested this, but I think you could say JobEnded True and set an end time, but make the batch job keep doing things afterwards and keep logging and it would still work (Is this true?)
The benefit it had was that new developers would look at the service and realize they needed to tell the logs when the batch job ended. So it encouraged them to write summaries at the end and be aware of when it stops in their design.
If we do remove it, we would have to remove the End Time on the batch job history widget from the IDE then. Unless EASYProcess would be able to tell when it ended by the fact that it stopped running?
The duration logic on that IDE page, if it used the end time, would need updates maybe
Message/AppendMessage/ResetMessage: Keep
I saw that in the new "WriteHistory" service, there is both a "Message" and "ResetMessage".
I think the meaning on these have shifted from the "UpdateStatus" service:
I think EASYCommerce is the main base app that uses batch jobs. Only real rare case we had for the override feature was by Cameron. He did it for the processed count though. I get that its a feature offered though and we might not want to remove features, so its fine to keep.
ProcessedCount/AddProcessedCount/ResetProcessedCount: Keep
definitely used
LastProcessed: Worth discussing
only used in matt m's design from easyprocess 4.8. Though this past year, the recommended batch job design failed and the internal team suggested to use the last processed count. So it did become relevant again recently.
It works by running the batch job to import 100 records, then end. When it runs, each record it records what its last processed record was. Then the dev schedules the batch job to run every minute or something like that, real often. When it starts again, it references the last time it ran to know where to start from.
UploadCount:can be removed
not used. this either falls into downloaded count which is processed, or the one in the source which is the estimate. We never used this in EC
Estimate: Keep
This is used in every batch job in EC, its a nice to have feature
ErrorCount: Keep
needed for error handling even in simple cases. We shouldn't assume or encourage a dev to have no error handling
Status: Worth discussing
We didnt have this in 5.1 yet. It was added in the service, but a feature request was made to add it to the Batch job history UI in the IDE and that was never completed because it wasn't priority. So its worth noting that we lived without this feature for a long time.
However, it was used in 4.8. because the EP_Imports table we used to record batch job history had a status column. So we could look down the records and see:
Complete
Complete
Complete
Error
Complete
And real fast you would know that a batch job failed earlier in the week.
Now you have to look at the processed count. So you would see something like:
2272
2272
2268
2269
492
2267
Here you would see that the count that we are importing is changing slightly. Then one day, with the 492, it failed. You can tell this because the counts were somewhat consistent before and after it. But its harder to tell if your talking about data where drastic changes are made:
86
172
56
48
The above could be a perfectly valid case for success. That 172 day could be the one day records overlapped and it got double the usual amount. There would be no errored batch job record, but you would have to click into each history record to confirm that.
Update Status:
Option 1: WriteHistory is a basic use case service
If its just the one you use for basic case and the UpdateStatus weren't deprecated, I'd be okay with that. Though there are still some changes it should have.
It needs ErrorCount to be added. Because we should never promote for the developer to not do their error handling
It doesnt need the option to reset messages and counts because those are advanced use cases. throwing out the data youve collected so far is a little dangerous and shouldnt be encouraged unless the developer knows what they're doing
So Write status would have the following inputs
- Message
- AddProcessedCount
- AddErrorCount
In this case, its worth looking at the existing inputs for the UpdateStatus and how those are used today:
Name: can be removed
This can be removed if EASYProcess knows which batch job is running. We don't have any complex need to update another batch job from the one that is running.
EndTime/JobEnded: Worth discussing
I think that the end time/JobEnded flag doesn't do anything. I never tested this, but I think you could say JobEnded True and set an end time, but make the batch job keep doing things afterwards and keep logging and it would still work (Is this true?)
The benefit it had was that new developers would look at the service and realize they needed to tell the logs when the batch job ended. So it encouraged them to write summaries at the end and be aware of when it stops in their design.
If we do remove it, we would have to remove the End Time on the batch job history widget from the IDE then. Unless EASYProcess would be able to tell when it ended by the fact that it stopped running?
The duration logic on that IDE page, if it used the end time, would need updates maybe
Message/AppendMessage/ResetMessage: Keep
I saw that in the new "WriteHistory" service, there is both a "Message" and "ResetMessage".
I think the meaning on these have shifted from the "UpdateStatus" service:
| Service | Paremeter to add to the existing message | Parameter to wipe the previous message and start new |
|---|---|---|
| WriteHistory | Message | ResetMessage |
| UpdateStatus | AppendMessage | Message |
ProcessedCount/AddProcessedCount/ResetProcessedCount: Keep
definitely used
LastProcessed: Worth discussing
only used in matt m's design from easyprocess 4.8. Though this past year, the recommended batch job design failed and the internal team suggested to use the last processed count. So it did become relevant again recently.
It works by running the batch job to import 100 records, then end. When it runs, each record it records what its last processed record was. Then the dev schedules the batch job to run every minute or something like that, real often. When it starts again, it references the last time it ran to know where to start from.
UploadCount:can be removed
not used. this either falls into downloaded count which is processed, or the one in the source which is the estimate. We never used this in EC
Estimate: Keep
This is used in every batch job in EC, its a nice to have feature
ErrorCount: Keep
needed for error handling even in simple cases. We shouldn't assume or encourage a dev to have no error handling
Status: Worth discussing
We didnt have this in 5.1 yet. It was added in the service, but a feature request was made to add it to the Batch job history UI in the IDE and that was never completed because it wasn't priority. So its worth noting that we lived without this feature for a long time.
However, it was used in 4.8. because the EP_Imports table we used to record batch job history had a status column. So we could look down the records and see:
Complete
Complete
Complete
Error
Complete
And real fast you would know that a batch job failed earlier in the week.
Now you have to look at the processed count. So you would see something like:
2272
2272
2268
2269
492
2267
Here you would see that the count that we are importing is changing slightly. Then one day, with the 492, it failed. You can tell this because the counts were somewhat consistent before and after it. But its harder to tell if your talking about data where drastic changes are made:
86
172
56
48
The above could be a perfectly valid case for success. That 172 day could be the one day records overlapped and it got double the usual amount. There would be no errored batch job record, but you would have to click into each history record to confirm that.