1. Market Surveillance Systems

* Market:
    * Equtities
    * Options
    * Fixed Income
* Events (more than 37 billion):
    * orders
    * quotes
    * trades
* Patterns
    * Late trade
    * Late large trade
    * Paring trade
    * Park trade

2. SONAR (Securities Observations News Analytics and Regulatory)

Inside trading / fraud surveillance System that based on news. This project mainly focuses on cloud migration and enhance new features.

* System architecture
    ``Load balancer -> EC2 -> Elastic Search Cluster ->postgressSQL (for index) and S3 (meta data)``

* URL design:
    ``api.sonar.finra.org/fda/v1/indvl_pk (filter and pagingation_data)``
* Exception design: proper status code returned
* Filters: logging / request forgery protection
* Interceptors: manipulate entity (input / output)

3. Json Validator

com.github.fge

4. MarketSpace

* Easy relation: exceptions -> alert (Firm)
* Pattern surveillance results
* System architecture
    ``Angular 2 -> spring -> postgressSQL``

5. Pattern Surveillance

* S3 storage: daily / monthly
* EMR: Elastic MapReduce (spot / on-demand instances)
* Hive
    * Use HDFS as storage
    * Use MapReduce for parallel processing
    * Hive Driver/Complier: execute hiveSQL and generate execution plan
    * Hive Meta store: partitions / tables / HDFS file / Data location
    * Use UDF(user define function) to solve null and date issues
* Data size: 1 month / 100 billion rows / 5 TB / 100 general instances / 3 H 12 mins / $57