The test SF = 10000, stream number Sq = 4.

NOTE: Index is not used in the Transwarp TPC-DS test.

The create_text_tables.sql is from v2.11.0rc2/tools/tpcds.sql with some changes to meet the Spec
1) varchar is changed to varchar2: VARCHR2 in Transwarp is the traditional VARCHAR to meet
 "Section 3.4.5 Empty column values, as generated by dsdgen, must be treated as NULL values in the data processing system,
 i.e. the data processing system must be able to retrieve NULL-able columns using 'is null' predicates."
2) The type of *_number and *_quantity are changed from "int" to "bigint", since in 10T test, the value is out of the range of int.

The sql just adds a schema to the raw data on hdfs. The DB name is tpcds_text_10000.

TEXT DB is just a bridge to load from raw text data, which does not reload the data.

HOLODESK DB is DB for test. The Clause_3/doLoad.sh will create HOLODESK tables and load data from the text tables.


dsdgen is wrapped as map/reduce program for large scales on a hadoop cluster.

To generate the data for scale = 10000,

 hadoop jar ./target/tpcds-dsdgen-2.11.jar -Ddfs.blocksize=4g -Ddfs.replication=2 -d /user/beeline/tpcds/10000/ -s 10000 -p 2111

Notes: Due to disk size limitation, we set 2 replications for raw data on HDFS.

The codes is from TPC-DS tool kit without change.

The raw data is at HDFS: /user/beeline/tpcds/10000

hdfs dfs -dus /user/beeline/tpcds/10000
9897109626018  /user/beeline/tpcds/10000

hdfs dfs -dus -h /user/beeline/tpcds/10000
9.0 T  /user/beeline/tpcds/10000


tpcds_holodesk_10000 is the DB name in test. The test tables are in HOLODESK format.

Fact tables: *_sales, *_returns and inventory are designed with date-based partitions. Each partition is a month or quarter.
Note: For partition table, the partition column must be the last column in the table.

Row numbers after load:
call_center:		54 
catalog_page: 		40000
catalog_returns:	1,440,033,112
catalog_sales:  	14,399,964,710 
customer:		65000000
customer_address:	32500000
customer_demographics:	1920800
date_dim:		73049
household_demographics:	7200
income_band:		20
inventory: 		1,311,525,000
item:			402000
promotion:		2000
reason:			70
ship_mode:		20
store:			1500
store_returns:		2,879,512,807
store_sales: 		28,799,795,112
time_dim:		86400
warehouse:		25
web_page: 		4002
web_returns:		720,020,485
web_sales:  		7,199,963,324
web_site:		78


After load, the table data will be put in shiva, which is configured to use 3 replications by default.
From user's view, we donot know which data is on which disk. 

After load, we can read the talbe sizes in inceptor-server-argodbcomputing1 UI:
10.216.91.101:8888/argodb/set4040 -> holodesk, then select a table, it will show the details which include the TotalSize. 


--Table schema--
The table description output is "table-schema.log". You can get it by the following command:
beeline -u jdbc:hive2://localhost:10000  --maxWidth=1000 -f desc_tables.sql 

Please note: The partition column is the last column of the table.
