Page 1 of 1

Solr/EASYCommerce Catalog Setup

Posted: May 23rd, 2023, 1:37 pm
by CathyC
For EASYCommerce, the categories of items can be maintained using Solr.
I will note that many applications are built to perform the same logic using application database queries and so there is a configuration variable (Catalog_ItemCatalogSource) which can be set to "Solr"/"Application Db".
ItemCatalogSource Configuration Variable.PNG
ItemCatalogSource Configuration Variable.PNG (10.02 KiB) Viewed 1437 times
ItemCatalogSource Configuration Variable.PNG
ItemCatalogSource Configuration Variable.PNG (10.02 KiB) Viewed 1437 times
Though even if using the Application Db source, it uses all the same areas of the application, they just run different logic. The only portion that would not be used would be the Solr Schema file.

Below is an explanation of how this system works using Solr to perform quick searches and support complex custom catalogs or a basic design, but be built for growth.

This includes the setup of:
  • Item Import
  • Reindex
  • Catalog
  • Catalog Search Method
  • Catalog Left Panel/Category Facet Filters
  • Site Header Category Menu/Category Facet Menu
  • Solr Configuration File
Item Import
ItemImport Batch Job Screenshot.PNG
ItemImport Batch Job Screenshot.PNG (6.16 KiB) Viewed 1437 times
ItemImport Batch Job Screenshot.PNG
ItemImport Batch Job Screenshot.PNG (6.16 KiB) Viewed 1437 times
The item import is meant to import the items from JDE into the application database tables.
typically we populate tables meant to simulate the table structure in JDE, but could have as many as below:
  • EC_Item_Master
  • EC_Item_Branch
  • EC_Item_Account
  • EC_Item_Media
  • EC_Item_Catalog
If others are needed, they follow the same naming convention and the item import batch job needs to be edited to populate it with info from other integration points, typically JDE.


Reindex
Reindex Method Screenshot.PNG
Reindex Method Screenshot.PNG (9.06 KiB) Viewed 1437 times
Reindex Method Screenshot.PNG
Reindex Method Screenshot.PNG (9.06 KiB) Viewed 1437 times
Solr stores the items it will be searching, but its collection of items needs to be synced with the application database records at a regular basis.
Typically at the end of the item import, a "Reindex" service will run.
That service accepts a query. This is the query that joins the application database tables to create the data set that is given to Solr.

Typically there will be a method called something like "ItemCatalogQuery" which outputs the query used to create the catalog selection. The data set will exactly match what the users see on the catalog page and how searches are performed against this, so its important to get this part right or it may require changes later on.

This is built in a method because all the services that work with Solr need this query.
Sample ItemCatalogQuery.txt
(1.43 KiB) Downloaded 235 times
Sample ItemCatalogQuery.txt
(1.43 KiB) Downloaded 235 times
METHOD-10000195 - ItemCatalogQuery.xml
(170.04 KiB) Downloaded 217 times
METHOD-10000195 - ItemCatalogQuery.xml
(170.04 KiB) Downloaded 217 times

Catalog
Catalog Screenshot.PNG
The catalog page is a fairly simple page. It is set to call some method and display the items it returns on the page in a grid or a list. Improvements can be made to this page to display them in complex ways, but in a simple form, the catalog itself is not complex.

Some examples of controls that can be added to the catalog page that call the controller:
  • Results per page: 10, 30, 60, etc
  • "Filter Results By" search. Where keywords can be typed in which filter the current selection
  • Sorting by fields ASC/DESC
  • Paging. Viewing page 1, 2, 3, ... 50. Which uses some math from the "results per page" to find out what the start and end number of the catalog should be. The paging buttons and that math can get a little confusing, so there is an export of that logic we have to start with
  • Category Filters. "Show only the products in this product family. This is called the catalog left menu and will be explained later.
METHOD-10000198 - ItemCatalogPagingLinks.xml
(143.96 KiB) Downloaded 208 times
METHOD-10000198 - ItemCatalogPagingLinks.xml
(143.96 KiB) Downloaded 208 times

Catalog Search Method
The catalog search method receives input of what type of search is being performed. The added controls to the catalog page end up adding more filters (like showing 30 vs 60 results per page, sorting, showing results on page 50, etc). So at the very beginning of this method, it stores all the types of filters it could receive and what values it is given to know what search it will perform.

The catalog search method also calls a Solr service so it too needs to call the "ItemCatalogQuery" method to have this value to pass into the service.

From the inputs to the Catalog Search Method, a search query is created. But the query is in Solr syntax since it will be passed into a Solr service called "Search".
Here is a typical Solr Search Service query:
Solr Service Search Value.PNG
This is performing a search that says "show me all items AND ones in the catalog called "Default" AND ones available for order in branch plant "M30" AND ones that are either specifically available to me at address 4242 OR available to anyone with no restrictions"
In Solr, the "*" is a wildcard character.

Once results return from the Solr search service, the rest of the Catalog search method's job is to get the extra bit of info that Solr was not in charge of holding, but that should be displayed on the catalog. This can include, price, availability, product image.

We put the Catalog Search method in charge of this because then this method can be used from anywhere on the application to search for items. We would want to do this because recreating it elsewhere would risk not using the same search criteria and could accidentally show restricted items to the end user.
Then if everywhere that performs search is using the same method, we want the same stuff to display everywhere we show the item. so it makes sense to also need the image, price, etc in these secondary locations. So instead of duplicating the effort to get that extra info, we put it in the catalog search method to be done once.
METHOD-10000155 - ItemCatalogSearch.xml
(1.27 MiB) Downloaded 206 times
METHOD-10000155 - ItemCatalogSearch.xml
(1.27 MiB) Downloaded 206 times

Catalog Left Panel/Category Facet Filters
The left panel of the catalog displays the categories that exist for the items on the page.
Catalog Left Menu Screenshot.PNG
Catalog Left Menu Screenshot.PNG (2.73 KiB) Viewed 1437 times
Catalog Left Menu Screenshot.PNG
Catalog Left Menu Screenshot.PNG (2.73 KiB) Viewed 1437 times
This is also produced from the output of the "Catalog Search" method and its Solr "Search" service. To do this, the Solr Search service receives an input of which Categories it should watch for. Solr calls these Facets. In the input, the service will be given a value like "CategoryCode1Value" and "CategoryCode2Value" and then for all the items it returns, it keeps a tally of how many of various types of those fields are found.

Here is an example of the output it returns in the Solr "Search" service if using the FacetColumns:
Catalog Search Service FacetColumns Output in Logs.PNG
This is nearly ready for use, but it only contains the values. "ABS" cant be shown to the user because they need to see the category code description of "Abrasives". So a loop is done to find the description for all the distinct values and merge it back to the original value.

The "Catalog Search" method just outputs this XML to be used by method: "ItemCatalogFacetsMenu". This takes the XML of the various Facets for the returned items and builds the html to create the hyperlinks that show in the left panel.

Here is the XML export that creates this
METHOD-10000196 - ItemCatalogFacetsMenu.xml
(122.26 KiB) Downloaded 259 times
METHOD-10000196 - ItemCatalogFacetsMenu.xml
(122.26 KiB) Downloaded 259 times
Once this method creates the HTML, it is put on the page in an HTMLGenericControl


Site Header Category Menu/Category Facet Menu
EC Category Menu Example Screenshot.PNG
EC Category Menu Example Screenshot.PNG (10.89 KiB) Viewed 1437 times
EC Category Menu Example Screenshot.PNG
EC Category Menu Example Screenshot.PNG (10.89 KiB) Viewed 1437 times
At the top of the page is the header that shows on every page. This can be changed to have a menu that shows the categories for all items in the catalog (that the user has access to).
This would allow the user to browse the categories and filter down until they see a subset of the items that they wish to browse.

This is done using another Solr service "GetFacetedValues". This is another Solr service, so again it needs the full query that gets passed into all Solr services. So we need to call "ItemCatalogQuery" method again and pass that as input.
It also needs to know which facets we are interested in. Maybe when the items show on the catalog page, we show all possible facets: catcod1, catcode2, catcode3, catcode4. But when shown in the menu, we only want to show 2 levels deep. So we would enter those 2 values: Catcode2, catcode2.
And finally the Solr service again needs the Solr query search value. This should be identical to the search performed in the CatalogSearch Method. If it is not, you risk showing items the user is not allowed to see.

Once the service runs, it produces an output like this:
GetFacetedValues Logs Screenshots.PNG
This says that there are 343 items in total under the categorycode1 value of "JEA" which has description "Jeans".
Of those, 343 of them (all) belong to the categorycode2 value of "WI" which has the description "Waist".

Now, not all of this info came from the Solr service. Some of it was found afterwards and merged back to the service's xml so it can be used to create the html of the menu that shows on the site.

Solr Configuration File
The part I wont explain how to set up is the solr configuration file. It sits on the server and is named managed-schema.
This cant be setup by the EASYProcess developer because this wont be accessible from EASYProcess due to security permissions.
The file stores the definitions of its datatypes and each column and what datatype it is set up as.
The query that populates Solr with data needs every field in its select clause to already exist in the Solr schema.
Solr Schema File Screenshot.PNG
Solr Schema File CopyField Screenshot.PNG
Solr Schema File CopyField Screenshot.PNG (9.03 KiB) Viewed 1437 times
Solr Schema File CopyField Screenshot.PNG
Solr Schema File CopyField Screenshot.PNG (9.03 KiB) Viewed 1437 times