Data loading guidelines

When creating Data reader protocols, it is recommended to create different protocols for loading master files and transactional data. This is a fairly common categorization, since master files usually contain information that feeds into Entities and Relationships (for example, the customer master file contains customer names, cities, states, sales representatives, area managers, and so on), and the transaction data usually contains information that feeds into Cubes (for example, Sales amount, Quantity, price, Discount, etc.).

The master files that feed into Entities and Relationships should be imported before importing the transaction files.  It is necessary to populate Entities and Hierarchies first and then Cubes so that aggregated versions (if any) can be correctly populated.

For example, consider a Sales amount Cube which has 2 versions, one organized by Customer-Product-Month (the primary version) and another one organized by City-Product-Month. The transaction file contains the following fields: date, customer code, product code, and sales amount. When reading this file, the secondary version can only be fed if the Customer→City relationship is already populated; it is, therefore, necessary to first read the Customer master file containing the Customer→City relationship.

It is also recommended to import only those Entity members that you actually need. For example, in a Board Data model where you need to load 3 years of historical data, only customers and products with transactions in that time interval are needed.

Optimization tips

• It is recommended to upload data in chronological order to obtain an optimized Data model

• When loading large amounts of data, significant improvements in loading performance can be achieved by sorting data by date first, and then sorting by the fields corresponding to Entities with the smallest Max item Number.
  For example, consider a scenario where we have a table with Date, Customer, Product, and Sales Amount as fields and Month, Product, and Customer as the corresponding Entities in Board. Assuming that the number of customers is greater than the number of products, the source data should be sorted by the Date-Product-Customer fields.
  If you are loading data through ODBC APIs, use the ORDER BY clause in the SQL statement to sort the data or to create an index in the table.

• If a Data reader exceeds the Max item number when loading Entity members, a warning message appears and the Data reader will discard all excess members from that point on. If the Data reader log has been enabled from the Data reader table, a log file of the rejected records for each Data reader will be created. Read more about the Data reader log.

Valid date formats

Board standard Time Entities accept the following date formats.