Scala Materials


Quick References
Week 1
!!!
!!!
Week 2
Week 3+

  • Scala By Example, for more examples which illustrate concepts covered in the lectures.
  • Scala Cheatsheet, for a quick reference covering pattern matching syntax, for-comprehension syntax, and more.

评论

发表评论

此博客中的热门博文

8 Link Analysis

图片
Pagerank—link analysis  Early web search: extract keywords-->queries match -->page ranked by  occurrences of keywords Term Spam: (TS) Disguise a page as sth it is not about  Pagerank: vs TS Rank pages by linkage too How many pages point to a page= # incoming links => importance Link spam (fade incoming pointers) Manually identify some important pages and assign them page ranks , then let ranks propagate to other nodes. Random surfer model  Random surfer start from any page follow its outgoing links randomly   来，去，都是随机的 Pagerank = Prob ( a random surfer lands on the page ) More robust than manual approach , a collective voting scheme Limiting distribution  Graph is strongly connected , a node can reach any other node in the graph But cannot have dead end & spider traps A--->B, C, D;    B--->A, D,(A,D->C);     C--->A(->B,C,D);   ...
继续阅读

1 Map reduce problems

图片
PROBLEMS Join   Matrix multiplication 2 / 1 phase  Matrix vector multiplication  POINTS Map reduce input output Memory requirement  Reducer size  Communication cost P1 Join:  R ( A , B )  S ( A , C )   1. Map r ( a , b ) => ( a , ( 'R' , b ));  s ( a , c ) => ( a , ( 'S' , c ))   2. Reduce ( a , [( 'R' , b ), ( 'S' , c1 ), ( 'S' , c2 )]) => ( a , ( b , c1 )), ( a , ( b , c2 ))   Dangling tuples: Key with values from only 1 relation  ( a , [( 'R' , b )]) => left dangling ; ( a , [( 'S' , c )]) => right dangling  P2 Matrix-multiplication  C mXn = A mXp B pXn  2 phase: a. phase1  product  Map : Extract key from A ’ s c , B ’ s r A [ i , k ] => ( k , ( 'A' , i , A [ i , k ])) ; B [ k , j ] => ( k , ( 'B' , j , B [ k , j ]))   Communication cost : mp + np  Reduce...
继续阅读

How to do addition for sparse vectors

How to add sparse vectors  http://stackoverflow.com/questions/32981875/how-to-add-two-sparse-vectors-in-spark-using-python Week 4  NoSQL Building large scalable web applications  platform—do analysis Programming note >>> zip ([ 1 , 2 ],[ 0 , 3 ]) [( 1 , 0 ), ( 2 , 3 )] >>> dict ( zip ([ 1 , 2 ],[ 0 , 3 ])) {1: 0 , 2: 3} Something like this should work: from pyspark.mllib.linalg import Vectors , SparseVector , DenseVector import numpy as np def add ( v1 , v2 ) :     """Add two sparse vectors     >>> v1 = Vectors.sparse ( 3 , {0: 1.0 , 2: 1.0} )     >>> v2 = Vectors.sparse ( 3 , {1: 1.0} )     >>> add ( v1 , v2 )     SparseVector ( 3 , {0: 1.0 , 1: 1.0 , 2: 1.0} )     """     assert isinstance ( v1 , SparseVector ) and isinstance ( v2 , SparseVector )     assert v1.size == v2.size...
继续阅读