Next: System Block Diagram Back to:
Next: Message Channel Next: Message Packing - eCore Next: Daemon Design The Controller is a Linux workstation running a Daemon
Back to:
Next: Application Back to:
Next: Local Memory Channel 1 Next: Local Memory Channel 2 Next: Distribution A simple job control system is required
Next: Message Types
Next: Packing Message Data
At the time of writing each major component of the system has been prototyped
Contributors or sponsors for this work are always welcome!
Essential interface functions for ARM and Epiphany software. Built separately with ARM (e-hal.h) and the Epiphany (e_lib.h) libraries so that application code can use the same interface in ARM code and Epiphany code. Used during the MapReduce operation as a workspace for received messages Back to:
Used to hold an interim linked list of data for eCore consumers
Back to:
Used to manage communications between eCore. Uses "Busy" (non-zero) "Free" arbitration.
Back to:
Used to pass commands between eCore. Uses InQueue for arbitration.
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Back to:
Designed to carry the command and arguments to the remote system and the results back to the caller Back to:
Designed to carry one of a number of different requests to the provider
Back to:
Bulk Synchronous Parallel: library to assist in running tasks in a parallel processing system Back to:
Map operates on a list of values in order to produce a new list of values, by applying the same computation to each value. The order in which the elements are accumulated can be important. Fold-left combines items in the sequence they were created. Fold-right goes in the other direction. Back to:
Map operates on a list of values in order to produce a new list of values, by applying the same computation to each value. The order in which the elements are accumulated can be important. Fold-left combines items in the sequence they were created. Fold-right goes in the other direction. Back to:
https://github.com/earonesty/grun A lightweight replacement for job queueing systems like LSF, Torque, condor, SGE, for private clusters. Back to:
Map Reduce is an architecture framework designed for creating and processing large volumes of data using clusters of computers. Google's paper Dean2008 MapReduce: Simplified Data Processing on Large Clusters is now often cited as the introduction of Map Reduce to the problem of data processing. Map operates on a list of values in order to produce a new list of values, by applying the same computation to each value. Reduce operates on the output of Map to produce the final result. The order in which the elements are accumulated can be important. Fold-left combines items in the sequence they were created. Fold-right goes in the other direction. Back to:
In the 16 core Epiphany chip the eCores are arranged in 4 rows and 4 columns. The rank of a given core is . This provides a convenient way to allocate different tasks to each eCore with a single number. Back to:
Back to:
@article{Dean:2008:MSD:1327452.1327492,
author = {Dean, Jeffrey and Ghemawat, Sanjay},
title = {MapReduce: Simplified Data Processing on Large Clusters},
journal = {Commun. ACM},
issue_date = {January 2008},
volume = {51},
number = {1},
month = jan,
year = {2008},
issn = {0001-0782},
pages = {107–113},
numpages = {7},
url = {http://doi.acm.org/10.1145/1327452.1327492},
doi = {10.1145/1327452.1327492},
acmid = {1327492},
publisher = {ACM},
address = {New York, NY, USA},
}
The user remotely launches the Application on the Parallella from the Controller. The Application manages the workload on the Epiphany eCore.
Next: Controller
Next: Shared Memory Channel
Back to:
char workspace[tbd];
unsigned InQueue[MAXRANK];
#define SETENTRY(o,r) \
((((unsigned)(o) << 4 * sizeof(unsigned)) & ORDMASK) + ((unsigned)(r) & RNDMASK))
typedef enum {
WL_IDLE = 0,
WL_CANCEL,
WL_CURR,
WL_SEQ, // numbers to 0xFFFE are available to sequence commands
WL_SYNC = 0xFFFF, // wait for all work to complete
} work_order_t;
char inqueuestore[MAXRANK][tbd];
msg_channel_t *msg_channel_store_cons;
msg_channel_t *msg_channel_store_prd[MAXRANK];
msg_channel_t *msg_channel_inqueue_cons;
msg_channel_t *msg_channel_inqueue_prd;
msg_channel_t *msg_channel_producer;
msg_channel_t *msg_channel_consumer;
msg_channel_t *msg_channel_core;
msg_goto_next_sgmnt(msg_channel_core);
int (*mr_map)(char **wkp, char *wklimit, el_t **mqnxtfree, char *mqlimit);
unsigned (*mr_allocate)(el_t *item, unsigned maxid);
int (*mr_mappack)(el_t *ucitem, char **msp, char* msplimit, size_t *size);
void (*mr_reduce)(el_t *usitem);
int (*mr_reportpack)(char **msp, char* msplimit, size_t *size);
int (*mr_mvuc)(el_t *ucitem)
int mr_emitus(el_t *usitem);
int mr_mapemit(el_t *ucitem);
typedef struct {
unsigned row; // row of remote eCore (or 0 for shm)
unsigned col; // col of remote eCore (or 0 for shm)
size_t initialoffset;
size_t offset;
size_t sgmntsize;
size_t limit;
size_t size;
void *mem; // Either &e_group_config or &emem
#ifdef __epiphany__
e_memseg_t emem;
#else
e_mem_t emem;
#endif
} msg_channel_t;