Page 1 of 1

UpdateStatus/WriteHistory Improvements

Posted: December 19th, 2022, 5:28 pm
by CathyC
I was talking with Cameron recently and realized the UpdateStatus is deprecated and replaced with WriteHistory which has way less inputs. The WriteHistory service should receive some improvements, but it depends how we plan to use it going forward. I saw 2 possible options and outlined them below

Update Status:
image.png
Write Status:
image.png

Option 1: WriteHistory is a basic use case service
If its just the one you use for basic case and the UpdateStatus weren't deprecated, I'd be okay with that. Though there are still some changes it should have.
It needs ErrorCount to be added. Because we should never promote for the developer to not do their error handling
It doesnt need the option to reset messages and counts because those are advanced use cases. throwing out the data youve collected so far is a little dangerous and shouldnt be encouraged unless the developer knows what they're doing

So Write status would have the following inputs
  • Message
  • AddProcessedCount
  • AddErrorCount
Option 2: WriteHistory is being updated to be the only service, with useless inputs removed
In this case, its worth looking at the existing inputs for the UpdateStatus and how those are used today:

Name: can be removed
This can be removed if EASYProcess knows which batch job is running. We don't have any complex need to update another batch job from the one that is running.

EndTime/JobEnded: Worth discussing
I think that the end time/JobEnded flag doesn't do anything. I never tested this, but I think you could say JobEnded True and set an end time, but make the batch job keep doing things afterwards and keep logging and it would still work (Is this true?)
The benefit it had was that new developers would look at the service and realize they needed to tell the logs when the batch job ended. So it encouraged them to write summaries at the end and be aware of when it stops in their design.
If we do remove it, we would have to remove the End Time on the batch job history widget from the IDE then. Unless EASYProcess would be able to tell when it ended by the fact that it stopped running?
The duration logic on that IDE page, if it used the end time, would need updates maybe


Message/AppendMessage/ResetMessage: Keep
I saw that in the new "WriteHistory" service, there is both a "Message" and "ResetMessage".
I think the meaning on these have shifted from the "UpdateStatus" service:

ServiceParemeter to add to the existing messageParameter to wipe the previous message and start new
WriteHistoryMessageResetMessage
UpdateStatusAppendMessageMessage
I think EASYCommerce is the main base app that uses batch jobs. Only real rare case we had for the override feature was by Cameron. He did it for the processed count though. I get that its a feature offered though and we might not want to remove features, so its fine to keep.


ProcessedCount/AddProcessedCount/ResetProcessedCount: Keep
definitely used

LastProcessed: Worth discussing
only used in matt m's design from easyprocess 4.8. Though this past year, the recommended batch job design failed and the internal team suggested to use the last processed count. So it did become relevant again recently.

It works by running the batch job to import 100 records, then end. When it runs, each record it records what its last processed record was. Then the dev schedules the batch job to run every minute or something like that, real often. When it starts again, it references the last time it ran to know where to start from.

UploadCount:can be removed
not used. this either falls into downloaded count which is processed, or the one in the source which is the estimate. We never used this in EC

Estimate: Keep
This is used in every batch job in EC, its a nice to have feature

ErrorCount: Keep
needed for error handling even in simple cases. We shouldn't assume or encourage a dev to have no error handling

Status: Worth discussing
We didnt have this in 5.1 yet. It was added in the service, but a feature request was made to add it to the Batch job history UI in the IDE and that was never completed because it wasn't priority. So its worth noting that we lived without this feature for a long time.

However, it was used in 4.8. because the EP_Imports table we used to record batch job history had a status column. So we could look down the records and see:
Complete
Complete
Complete
Error
Complete

And real fast you would know that a batch job failed earlier in the week.
Now you have to look at the processed count. So you would see something like:

2272
2272
2268
2269
492
2267

Here you would see that the count that we are importing is changing slightly. Then one day, with the 492, it failed. You can tell this because the counts were somewhat consistent before and after it. But its harder to tell if your talking about data where drastic changes are made:

86
172
56
48

The above could be a perfectly valid case for success. That 172 day could be the one day records overlapped and it got double the usual amount. There would be no errored batch job record, but you would have to click into each history record to confirm that.

Re: UpdateStatus/WriteHistory Improvements

Posted: December 20th, 2022, 12:17 pm
by CathyC
I talked with Suman on this and here are the notes of our talk:

The WriteHistory is intended to replace the old one, so we will need to make it for basic and advanced use case. A benefit of the change is to get to use more intuitive name that it writes to a log the history of the batch bjo

EndTime - in the IDE History widget, it sometimes doesnt have an end time. Might be though when its manually killed from the batch jobs ide page. Internal team will look into if the system can find this.

Last Processed - This actually still has a use case in the save to db. If we're imorting 100k and we get to 50k then errors. I dont want to restart the whole import if it takes a real long time. So i'd like to be able to start again at 50k. So keep

Estimate Count - would be good to have on the batch job history widget too. The main reason it was removed was because its an input of updatestatus that isnt on that widget in the IDE

ErrorCount - would be good to have on the batch job history widget too. The main reason it was removed was because its an input of updatestatus that isnt on that widget in the IDE

Status - would be needed in the batch job history widget. The main reason it was removed was because its an input of updatestatus that isnt on that widget in the IDE. Note this is different than the running/stopped on the main batch job ide page.

With these considerations, this is how the WriteHistory will look:

New Proposed Input:
  • AddToMessage
  • AddToProcessedCount
  • AddToErrorCount
  • EstimatedCount
  • LastProcessed
  • Status
  • EndTime/JobEnded?
  • ResetMessage
  • ResetProcessedCount
  • ResetErrorCount

Re: UpdateStatus/WriteHistory Improvements

Posted: December 20th, 2022, 1:40 pm
by CameronDouthitt
This sounds like everything I need to maintain the same kind of functionality that I am used to using for this.