Pagination - Composite Cache

Introduction

Office workers maintain various records with various page layouts and use cases. Some pages display an entire, small, reference table and the user selects and edit the appropriate record. Some have search criteria to list the desired records in pages and the user scrolls through each in a details page. Others let the user find a customer record and examine all manner of related data and logs in tabbed pages. Reading records takes time, so a web front-end should lazy load as needed and keep data in session.

Thus, at the heart of pagination is a Composite Cache, a configurable helper object stored in session extending AbstractCompositeCache, AbstractCompositeCache_1S and the like. These contain a master list and up to fifteen slave lists. Depending on configuration, each list usually has a client supplied list finder Command, a page finder Command and a single item finder Command for lazy loading detail of the selected record. The list finders for the slave list lazy loads records based on the selected record of the master list.

Client Design

Before writing Struts Actions, consider what lists need to be displayed and how they're found. For each list, consider the following aspects and config.

Composite cache aspects and config

Aspect	Config property	Notes
Name	name	Used in logs.
Base record	T generic type	Data Transfer Objects (DTO) that are the elements of the list and generally describe records of a single table.
Primary key	K generic type, keyExtractor	Configuration use an adapter to extract the primary key from the base record. This is normally a simple lambda function.
Other base record data		Records related to the base record shown in the list page, if any. For database performance, this should be restricted to DTOs of small, reference tables.
Page extension data	pageExtensionAssembler	If the list page expects data that is expensive to read, it might be worth using base record list pagination, see below..
Detail data	hasDetails	Data expected by the details page, if any, not loaded by list and page finders and must be lazy loaded by a single item finder.
Pagination mode	paginationMode	At this point, what data should be lazy loaded in pages, if at all, should be apparent. See below for pagination modes.
List/id/size finder	listFinder, idListFinder, or listSizeFinder	At this point, the client supplied eager loading Commnand should be written.
Page finder/ page extension assembler	pageExtensionAssembler, pageByIdsFinder, or pageByIndexRangeFinder	At this point, the client supplied lazy loading Commnand should be written.
Single item finder	itemFinder	At this point, the client supplied detail loading Commnand should be written. Even if details don't apply, it's best to provide this as Save List Actions may update a database record and request a reload of the selected item.

See an example below, which is a simplified version used in customer account pages.

Example composite cache

Slave list no	Name	Description	View List Action	Base record	Other base record data	Page extension data	Detail data	Pagination mode	List/id/size finder	Page finder/ page extension assembler	Single item finder
Master	Master	Customer accounts	{@link ViewAccountListAction} and {@link ViewAccountAction}	{@link GenericAccountDTO}	service, phone nos, and profile.		Personal numbers, IVR recordings, emails, VIP membership, and Wallet account	Full list or page by ids	Supplied by {@link FindAccountsInitialAction} and {@link FindAccountsMultiAction}	{@link MasterMultiItemFinder}	{@link MasterItemFinder}
1	Account history	Various audit records	Various	{@link CompositeHistoryDTO}				Full list	Supplied by various Struts actions
2	Subscriptions	Subscriptions to features	{@link ViewAccountSubscriptionListAction}	{@link MemberSubscriptionDTO}	Subscription state, subscription type		Payment details (if available from service)	Page by index range	{@link MemberSubscriptionListSizeFinder}	{@link MemberSubscriptionPageFinder}	{@link MemberSubscriptionItemFinder}

Pagination Modes

There are four pagination modes supported.

Full list

A client supplied Command eager loads an entire list, not in pages. The View List Action may or may not display the list in pages. This is only recommended for small result lists. The Template class is ListFinder, such as below.

public static class MasterListFinder implements ListFinder<NA,NotesFilterDTO> {

    @Override

    public List<NotesFilterDTO> getList(NA selectedMaster) throws Exception {

        MiscellaneousModel miscellaneousModel;

        List<NotesFilterDTO> result;

            

        miscellaneousModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(MiscellaneousModel.class, MiscellaneousModel.JNDI_BINDING_NAME);

        result = new ArrayList<>(miscellaneousModel.findAllNotesFilter());

        Collections.sort(result, new MasterItemComparator());   // Sort by display order

        return result;

    }

}

   

Base record list

A client supplied Command eager loads every base record of a list and another lazy loads additions to base records for the page requested by the View List Action. This is useful for large result sets using joined tables as querying just a single table base record is usually a much faster initial load. The Template classes are ListFinder and PageExtensionAssembler. For an example of ListFinder, see Full list section above. For an example of PageExtensionAssembler, see below.

public static class MasterPageExtensionAssembler implements PageExtensionAssembler<SearchStaffDTO> {

    @Override

    public void assemblePageExtensions(Collection<SearchStaffDTO> items) throws Exception {

        ChatRecruitModel chatRecruitModel;

        Map<Integer,SearchStaffDTO> searchStaffLookup;

        Map<Object, List<StaffProfileDTO>> groupedStaffProfiles;

        Collection<StaffProfileDTO> staffProfiles;

            

        searchStaffLookup = items.stream().collect(Collectors.toMap(s -> s.getOperatorId(), s -> s));

        chatRecruitModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(ChatRecruitModel.class, ChatRecruitModel.JNDI_BINDING_NAME);

        staffProfiles = chatRecruitModel.findStaffProfilesByOperatorIds(new ArrayList<>(searchStaffLookup.keySet()));

        groupedStaffProfiles = staffProfiles.stream().collect(Collectors.groupingBy(sp -> sp.getOperatorId()));

        for (StaffProfileDTO staffProfile: staffProfiles) {

            searchStaffLookup.get(staffProfile.getOperatorId()).setStaffProfiles(groupedStaffProfiles.get(staffProfile.getOperatorId()));

        }

    }

        

}

Page by ids

A client supplied Commmand eager loads the id of every record of a list and another lazy loads the records of the page requested by the View List Action. This is useful for large result sets where a query returning just primary keys can use an index to skip reading records themselves, and order doesn't matter. The Template classes are IdListFinder and PageByIdsFinder. See below.

public static class PartnerIdListFinder implements IdListFinder<NA,Integer> {

    private Integer partnerTypeId;

    private Integer status;

    private String partialName;

    private Integer partnerId;

    

    /**

     * Finds initial, empty list.

     */

    public PartnerIdListFinder() {

        this.partnerTypeId = PartnerTypeDTO.CLIENT;

        this.status = STATUS.ACTIVE;

        this.partialName = "";

        this.partnerId = null;

    }

    

    /**

     * Finds by form. 

     */

    public PartnerIdListFinder(Integer partnerTypeId, Integer status, String partialName, Integer partnerId) {

        this.partnerTypeId = partnerTypeId;

        this.status = status;

        this.partialName = (partialName != null)?partialName:"";

        this.partnerId = partnerId;

    }

    



    @Override

    public List<Integer> getIds(NA selectedMaster) throws Exception {

        PartnerModel partnerModel;

        int[] ids;

        String workingPartialName;

        int workingStatus, workingPartnerTypeId;

        

        partnerModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(PartnerModel.class, PartnerModel.JNDI_BINDING_NAME);

        if (partnerId != null) {

            return Arrays.asList(partnerId);

        } else {

            workingPartialName = (partialName != null)?partialName:"";

            workingStatus = (status != null)?status:0;

            workingPartnerTypeId = (partnerTypeId != null)?partnerTypeId:0;

            ids = partnerModel.getPartnerIdsBySql(workingPartialName, workingStatus, workingPartnerTypeId);

            return Arrays.asList(ArrayUtils.toObject(ids));

        }

    }

    ...

}



public static class PartnerPageFinder implements PageByIdsFinder<NA,Integer,PartnerDTO> {

    @Override

    public Collection<PartnerDTO> getItems(NA selectedMaster, Collection<Integer> keys) throws Exception {

        PartnerModel partnerModel;

        int[] ids;

        

        ids = ArrayUtils.toPrimitive(keys.toArray(new Integer[0]));

        partnerModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(PartnerModel.class, PartnerModel.JNDI_BINDING_NAME);

        return partnerModel.findPartnersByIds(ids, EnumSet.of(PartnerOption.DELETED, PartnerOption.PARENT, PartnerOption.PARTNER_TYPES));

    }

    

}

Page by index

A client supplied Commmand eager loads the size of a list and another lazy loads the records of the page requested by the View List Action by their indices in the list. This is useful for large result sets where the database can obtain the size of a result list and records using an index, rather than reading records beforehand. This is the hardest mode to implement as the order depends on the index used and only certain databases support such optimisation. See Querying Top-N Rows by Markus Winand for further details. The Template classes are ListSizeFinder and PageByIndexRangeFinder. Further, whereas the page finder in page by ids does not need to change, list size finders and page by index finders are linked in pairs. See examples below.

public static class SubscriptionsListSizeFinder implements ListSizeFinder2<StaffDTO> {

    private int operatorId;

    

    public SubscriptionsByProfileNameListSizeFinder(int operatorId) {

        this.operatorId = operatorId;

    }

    

    @Override

    public int getSize(StaffDTO selectedMaster) throws Exception {

        CustomerManagementModel customerManagementModel;

        int result;

        

        customerManagementModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(CustomerManagementModel.class, CustomerManagementModel.JNDI_BINDING_NAME);

        result = customerManagementModel.countMemberSubscriptionsByOperatorId(operatorId);

        return result;

    }



    public int getOperatorId() {

        return operatorId;

    }

}



public static class SubscriptionsPageByIndexRangeFinder implements PageByIndexRangeFinder<StaffDTO,MemberSubscriptionDTO> {

    private int operatorId;

    

    public SubscriptionsPageByIndexRangeFinder(int operatorId) {

        this.operatorId = operatorId;

    }

    

    @Override

    public List<MemberSubscriptionDTO> getItems(StaffDTO selectedMaster, int startIndex, int endIndex)

            throws Exception {

        CustomerManagementModel customerManagementModel;

        List<MemberSubscriptionDTO> result;

        

        

        customerManagementModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(CustomerManagementModel.class, CustomerManagementModel.JNDI_BINDING_NAME);

        result = customerManagementModel.getMemberSubscriptionPageByOperatorId(operatorId, endIndex - startIndex + 1, 

            startIndex, MemberSubscriptionOption.MEMBER_SUBSCRIPTION_STATE, MemberSubscriptionOption.MEMBER_SUBSCRIPTION_TYPE);

        return result;

    }

}

Single Item Finder

Single item finders finds a single record for a detail page, adding details list finders do not find. They are based on the Template classes SingleItemFinder. See an example below.

public static class SysRoleItemFinder implements SingleItemFinder<String,SysRoleDTO> {

    @Override

    public SysRoleDTO getItem(String key) throws Exception {

        SecurityModel securityModel;

        SysRoleDTO sysRole;

        

        securityModel = ServiceLocatorEJB3.getInstance().getInterfaceUnchecked(SecurityModel.class, SecurityModel.JNDI_BINDING_NAME);

        sysRole = securityModel.findSysRole(key);

        return sysRole;

    }

    

}

Implementation

Template class

Composite caches are a concrete implementation of AbstractCompositeCache, AbstractCompositeCache_1S etc. The example design found above leads to the following.

public class AccountCompositeCache extends AbstractCompositeCache_2S<

        GenericAccountId, GenericAccountDTO,

        CompositeHistoryId, CompositeHistoryDTO,

        MemberSubscriptionPK,MemberSubscriptionDTO> {

    ....

}

Constructor

First, write a default constructor and a copy constructor like the following.

public AccountCompositeCache() {

    super();

}



public AccountCompositeCache(AccountCompositeCache other) {

    super(other);

}

An instance should be stored in session, or rather in session per browser tab (see Session per Browser Tab), such as below.

private static final String ATTR_NAME_CACHE = AccountCompositeCache.class.getName() + "_CACHE";

    

public synchronized static AccountCompositeCache getInstance(BrowserTabSession browserTabSession) {

    AccountCompositeCache result;

    

    result = (AccountCompositeCache)browserTabSession.getAttribute(ATTR_NAME_CACHE);

    if (result == null) {

        result = new AccountCompositeCache();

        browserTabSession.setAttribute(ATTR_NAME_CACHE, result);

    }

    return result;

}

Config

Lastly, implement the Template functions for creating the initial configuration. These should be obvious from the design step, such as below.

@Override

protected ListCacheConfig<Object,GenericAccountId,GenericAccountDTO> getMasterListCacheConfig() {

    ListCacheConfig<Object,GenericAccountId,GenericAccountDTO> result;



    result = new ListCacheConfig<Object,GenericAccountId,GenericAccountDTO>();

    result.setHasDetails(true);

    result.setItemDetailChecker(new MasterItemDetailChecker());

    result.setItemFinder(new MasterItemFinder());

    result.setKeyExtractor(a -> a.getGenericAccountId());

    result.setListFinder(new MasterInitialListFinder());

    result.setName("master account");

    result.setPageByIdsFinder(new MasterMultiItemFinder());

    result.setPaginationMode(PaginationMode.FULL_LIST);

    return result;

}



@Override

protected ListCacheConfig<GenericAccountDTO, CompositeHistoryId, CompositeHistoryDTO> getSlaveListCache1Config() {

    ListCacheConfig<GenericAccountDTO, CompositeHistoryId, CompositeHistoryDTO> result;

    Calendar lastWeek;

    

    lastWeek = new GregorianCalendar();

    lastWeek.add(Calendar.DAY_OF_YEAR, -7);

    result = new ListCacheConfig<GenericAccountDTO, CompositeHistoryId, CompositeHistoryDTO>();

    result.setHasDetails(false);

    result.setKeyExtractor(item -> item.getCompositeHistoryId());

    result.setListFinder(new FindAccountHistoryAction2.HistoryFinderAll2(lastWeek.getTime(), null));

    result.setName("account history");

    result.setPaginationMode(PaginationMode.FULL_LIST);

    return result;

}



@Override

protected ListCacheConfig<GenericAccountDTO, MemberSubscriptionPK, MemberSubscriptionDTO> getSlaveListCache2Config() {

    ListCacheConfig<GenericAccountDTO, MemberSubscriptionPK, MemberSubscriptionDTO> result;

    

    result = new ListCacheConfig<GenericAccountDTO, MemberSubscriptionPK, MemberSubscriptionDTO>();

    result.setHasDetails(true);

    result.setItemDetailChecker(new MemberSubscriptionItemChecker());

    result.setIdListFinder(null);

    result.setItemFinder(new MemberSubscriptionItemFinder());

    result.setKeyExtractor(item -> new MemberSubscriptionPK(item.getPartnerId(), item.getId()));

    result.setListFinder(null);

    result.setListSizeFinder(new MemberSubscriptionListSizeFinder(false));

    result.setName("subscriptions");

    result.setPageByIdsFinder(null);

    result.setPageByIndexRangeFinder(new MemberSubscriptionPageFinder(false));

    result.setPaginationMode(PaginationMode.PAGE_BY_INDEX_RANGE);

    return result;

}

Example

The following example contains just a master list cache using full list pagination mode.

public class PrimeMinisterCompositeCache extends AbstractCompositeCache<Integer,PrimeMinisterDTO> {

    public static class PrimeMinisterItemFinder implements SingleItemFinder<Integer,PrimeMinisterDTO> {

        @Override

        public PrimeMinisterDTO getItem(Integer key) throws Exception {

            return PrimeMinisterTable.getInstance().findById(key);

        }

    }

    

    public static class PrimeMinisterListFinder implements ListFinder<NA,PrimeMinisterDTO> {

        @Override

        public List<PrimeMinisterDTO> getList(NA selectedMaster) throws Exception {

            return PrimeMinisterTable.getInstance().findAll();

        }

    }

    

    private static final String ATTR_NAME_CACHE = PrimeMinisterCompositeCache.class + "_CACHE";

    

    public synchronized static PrimeMinisterCompositeCache getInstance(HttpSession session) {

        PrimeMinisterCompositeCache instance;

        

        instance = (PrimeMinisterCompositeCache)session.getAttribute(ATTR_NAME_CACHE);

        if (instance == null) {

            instance = new PrimeMinisterCompositeCache();

            session.setAttribute(ATTR_NAME_CACHE, instance);

        }

        return instance;

    }

    

    

    public PrimeMinisterCompositeCache() {

        super();

    }

    

    public PrimeMinisterCompositeCache(PrimeMinisterCompositeCache other) throws Exception {

        super(other);

    }



    @Override

    protected ListCacheConfig<NA,Integer,PrimeMinisterDTO> getMasterListCacheConfig() {

        ListCacheConfig<NA,Integer,PrimeMinisterDTO> result;

        

        result = new ListCacheConfig<NA,Integer,PrimeMinisterDTO>();

        result.setHasDetails(false);

        result.setItemDeepCopier(item -> new PrimeMinisterDTO(item));

        result.setItemFinder(new PrimeMinisterItemFinder());

        result.setItemSorter(null);

        result.setKeyExtractor(g -> g.getId());

        result.setListFinder(new PrimeMinisterListFinder());

        result.setName("UK Prime Ministers");

        result.setPaginationMode(PaginationMode.FULL_LIST);

        return result;

    }



}