Unexpected loop behavior

[CathyC]

Akshay and I were working on this EC app and have an address import that pulls in account numbers (web, bill, ship), then checks if they have an addressid in the addressbook app db table. if they dont, it calls a getnextnumber to assign it and moves onto the next batch.

For this logic, I have a triple nested loop. Maybe that is the issue?
AddressImport batch job -> MainImport submethod -> LOOP136->LOOP145->LOOP146

Here is the walkthrough of my troubleshooting

Here I have the LOOP136 which is responsible for inserting the addresses into the addressbook table

In the logs we can see the Address service (CreateServiceFromXml) stores all the address info. by the end of this logic, it should use the insertandupdatenode service named UpdateWithAddressId to give each Address/Output/Result an AddressId

Here is an address where that worked:

Heres an address where that didnt work. See the blank AddressId node at the bottom.
So this is ERPAddressNumber 728601 with type BillTo.

so to look into how the addressid is supposed to be assigned, we will look in LOOP145.
Here is the logic canvas for LOOP145

You can see this canvas is given many addresses at a time from the parent LOOP. It then runs them through the CheckForRecords query against the AddressBook table. This is to try and find an existing AddressId. If none are found, it directs to the binary "MissingIds" which asks if any of them are missing their ids. If they are, it will enter a new LOOP146 to assign new AddressIds to those that are missing them.

We can see the CheckForRecords didnt find any matches, because all the addresses are new, so all the addresses would have been sent to the next nested loop

so looking at the next LOOP146, we can see its a real simple logic canvas. It gets one record at a time and calls the GetNextNumber service, then merges that new AddressId back to the LoopNode that was passed in.

Here we can see the logs from LOOP146 for the ERPAddressNumber 728601/BillTo
We can see that it does get the AddressId matched back to it

This confirms LOOP146 is working correctly for this record.
So we will start to look in the outer loops again for the problem.

Looking back at LOOP145, the node passed into the loop was:
WorkData/Prevalues/Output/Result/Address[AddressId='' or not(AddressId)]

So if we look in the logs in LOOP145 for Prevalues, we see that ERPAddressNumber 728601/BillTo did not get an AddressId.
Here is the root cause. It should have updated this record, like it did others but didnt

right after the LOOP146 call, it directs to the CreateServiceNodeFromLoop named LoopOutput.
This LoopOutput service is in a service-level for-each on node: WorkData/Prevalues/Output/Result/Address
So I would expect this not to work now.

But looking at the logs for LoopOutput, this service doesnt return anything to say that it was successful

so we'll have to look another level back.

Looking at LOOP136, it will have in the logs a "LoopOutput" node as if there was a service named "LoopOutput" on the canvas. This is what the CreateServiceNodeFromLoop does. It makes a node in the loop's parent.
So looking at the logs of LOOP136, we can see the LoopOutput service for ERPAddressNumber 728601/BillTo.
The LoopOutput logs in LOOP136 is missing the AddressId
This is how we expected since the Prevalues in LOOP145 also didnt have the value

The interesting part is that this is not always failing
Also interesting because based on the logs it looks like it should have succeeded

When running the address import in DV, I ran it a bunch of times, each time the error count was less: 1231, 839, 11, 0, 0, 0. Once all are imported, there are no longer any that fail, but that would be because it no longer takes this path to assign new ids.

When we promoted to QA, we had the a similar experience where we had to run it a bunch to get all records imported:
Final Processed Count: 6921
Error Count: 4614, 3276, 2693, 2201, 2145, 2142, 2142
But this one eventually stopped making progress. not sure why