Tuesday, April 18, 2017

K-Nearest Neighbours


http://www.geeksforgeeks.org/k-nearest-neighbours/
K-Nearest Neighbours is one of the most basic yet essential classification algorithms in Machine Learning. It belongs to the supervised learning domain and finds intense application in pattern recognition, data mining and intrusion detection.
It is widely disposable in real-life scenarios since it is non-parametric, meaning, it does not make any underlying assumptions about the distribution of data (as opposed to other algorithms such as GMM, which assume a Gaussian distribution of the given data).
We are given some prior data (also called training data), which classifies coordinates into groups identified by an attribute.
If we plot these points on a graph, we may be able to locate some clusters, or groups. Now, given an unclassified point, we can assign it to a group by observing what group its nearest neighbours belong to. This means, a point close to a cluster of points classified as ‘Red’ has a higher probability of getting classified as ‘Red’.
Intuitively, we can see that the first point (2.5, 7) should be classified as ‘Blue’ and the second point (5.5, 4.5) should be classified as ‘Red’.
Algorithm
Let m be the number of training data samples. Let p be an unknown point.
  1. Store the training samples in an array of data points arr[]. This means each element of this array represents a tuple (x, y).
  2. for i=0 to m:
      Calculate Euclidean distance d(arr[i], p).
  3. Make set S of K smallest distances obtained. Each of these distances correspond to an already classified data point.
  4. Return the majority label among S.
K can be kept as an odd number so that we can calculate a clear majority in the case where only two groups are possible (e.g. Red/Blue). With increasing K, we get smoother, more defined boundaries across different classifications. Also, the accuracy of the above classifier increases as we increase the number of data points in the training set.
struct Point
{
    int val;        // Group of point
    int x, y;       // Co-ordinate of point
    double distance;  // Distance from test point
};
 
// Used to sort an array of points by increasing
// order of distance
bool comparison(Point a, Point b)
{
     return (a.distance < b.distance);
}
 
// This function finds classification of point p using
// k nearest neighbour algorithm. It assumes only two
// groups and returns 0 if p belongs to group 0, else
// 1 (belongs to group 1).
int classifyAPoint(Point arr[], int n, int k, Point p)
{
    // Fill distances of all points from p
    for (int i = 0; i < n; i++)
        arr[i].distance =
            sqrt((arr[i].x - p.x) * (arr[i].x - p.x) +
                 (arr[i].y - p.y) * (arr[i].y - p.y));
 
    // Sort the Points by distance from p
    sort(arr, arr+n, comparison);
 
    // Now consider the first k elements and only
    // two groups
    int freq1 = 0;      // Frequency of group 0
    int freq2 = 0;      // Frequency of group 1
    for (int i = 0; i < k; i++)
    {
        if (arr[i].val == 0)
            freq1++;
        else if (arr[i].val == 1)
            freq2++;
    }
 
    return (freq1 > freq2 ? 0 : 1);
}



No comments:

Post a Comment

Labels

LeetCode (1243) GeeksforGeeks (1129) Review (896) Algorithm (764) LeetCode - Review (722) to-do (636) Dynamic Programming (340) Classic Algorithm (313) Classic Interview (266) Google Interview (246) Tree (146) POJ (141) Difficult Algorithm (129) LeetCode - Phone (127) EPI (125) Bit Algorithms (120) Different Solutions (117) Math (115) Lintcode (113) Cracking Coding Interview (110) Smart Algorithm (105) DFS (93) Binary Search (92) HackerRank (89) Greedy Algorithm (84) Binary Tree (83) Stack (78) Graph Algorithm (76) Two Pointers (64) BFS (63) LeetCode - Extended (62) Interview Corner (61) List (57) Advanced Data Structure (56) Geometry Algorithm (56) Interval (54) Codility (53) ComProGuide (52) LeetCode Hard (50) Trie (49) Union-Find (48) Binary Search Tree (47) Segment Tree (47) Algorithm Interview (46) USACO (46) Space Optimization (43) Bisection (42) Mathematical Algorithm (42) ACM-ICPC (41) Data Structure (40) Knapsack (40) Matrix (40) Time Complexity (40) Jobdu (39) Priority Queue (39) Recursive Algorithm (39) Sliding Window (39) Backtracking (38) String Algorithm (38) TopCoder (38) Codeforces (36) Introduction to Algorithms (36) Must Known (36) Beauty of Programming (35) Sort (35) Data Structure Design (34) prismoskills (33) Array (32) LeetCode - DP (32) HDU (31) Random (31) Google Code Jam (30) Graph (30) Permutation (30) Puzzles (30) Array O(N) (29) Company-Airbnb (29) Palindrome (29) Company-Zenefits (28) Pre-Sort (28) to-do-must (28) Logic Thinking (27) Microsoft 100 - July (27) Monotonic Stack (27) Queue (27) Binary Indexed Trees (26) Company - LinkedIn (25) Follow Up (25) GeeksQuiz (25) Post-Order Traverse (25) hihocoder (25) Company-Facebook (24) Algorithm Game (22) Hash (22) High Frequency (22) Merge Sort (22) DFS + Review (21) Lintcode - Review (21) O(N) (21) Brain Teaser (20) CareerCup (20) Company - Twitter (20) BST (19) Hard (19) Topological Sort (19) UVA (19) Sweep Line (18) Company-Uber (17) Game Theory (17) Left and Right Array (17) Probabilities (17) Proof (17) Codercareer (16) DP - Tree (16) Heap (16) Iterator (16) Shortest Path (16) String Search (16) Tree Traversal (16) itint5 (16) Difficult (15) KMP (15) Reverse Thinking (15) LeetCode - DFS (14) Number (14) Number Theory (14) Rolling Hash (14) TreeMap (14) Amazon Interview (13) Basic Algorithm (13) Codechef (13) Computational Geometry (13) Euclidean GCD (13) LeetCode - Classic (13) Long Increasing Sequence(LIS) (13) Majority (13) mitbbs (13) Algorithm - How To (12) Combination (12) Fast Power Algorithm (12) LeetCode - Thinking (12) Modify Tree (12) Reconstruct Tree (12) Reservoir Sampling (12) Simulation (12) 尺取法 (12) AOJ (11) Bucket Sort (11) DFS+Backtracking (11) DP - Interval (11) DP-Space Optimization (11) Divide and Conquer (11) Graph DFS (11) LCA (11) Math-Divisible (11) Miscs (11) O(1) Space (11) Prefix Sum (11) Princeton (11) X Sum (11) 挑战程序设计竞赛 (11) Coin Change (10) Company - Microsoft (10) DFS+Cache (10) DP - Bit Masking (10) DP - Digit (10) Facebook Hacker Cup (10) HackerRank Easy (10) Interval Tree (10) Kadane - Extended (10) MinMax (10) SPOJ (10) Theory (10) Tutorialhorizon (10) DP - Probability (9) DP-Multiple Relation (9) Mathblog (9) Max-Min Flow (9) Monotone Queue (9) Quick Sort (9) Stack Overflow (9) Stock (9) System Design (9) Use XOR (9) Book Notes (8) Bottom-Up (8) Company-Amazon (8) DFS+BFS (8) Dijkstra (8) LeetCode - TODO (8) Linked List (8) Longest Common Subsequence(LCS) (8) Prime (8) Quick Select (8) Suffix Tree (8) Summary (8) Tech-Queries (8) Traversal Once (8) TreeSet (8) 穷竭搜索 (8) Algorithm Problem List (7) Expression (7) Facebook Interview (7) Fast Slow Pointers (7) Fibonacci Numbers (7) Game Nim (7) Graph BFS (7) HackerRank Difficult (7) Hackerearth (7) Inversion (7) Kadane’s Algorithm (7) Level Order Traversal (7) One Pass (7) Pruning (7) Radix Sort (7) Tree DP (7) 蓝桥杯 (7) Bit Mask (6) Catalan Number (6) Classic Data Structure Impl (6) DFS+DP (6) DP - Monotone Queue (6) DP-Print Solution (6) Dutch Flag (6) Flood fill (6) How To (6) Interviewstreet (6) Knapsack - MultiplePack (6) Manacher (6) Minimum Spanning Tree (6) Morris Traversal (6) Multiple Data Structures (6) Pre-Sum (6) Programming Pearls (6) Rabin-Karp (6) Randomized Algorithms (6) Sampling (6) Schedule (6) Stream (6) Suffix Array (6) Xpost (6) n00tc0d3r (6) reddit (6) AI (5) Algorithm - Brain Teaser (5) Art Of Programming-July (5) Big Data (5) Brute Force (5) Code Kata (5) Codility-lessons (5) Coding (5) Company - WMware (5) Convex Hull (5) Crazyforcode (5) Cycle (5) DP - Knapsack (5) DP-Include vs Exclude (5) Find Rule (5) Graph Cycle (5) Hash Strategy (5) Immutability (5) Java (5) Matrix Chain Multiplication (5) Maze (5) Microsoft Interview (5) Quadtrees (5) Quick Partition (5) Quora (5) SPFA(Shortest Path Faster Algorithm) (5) Subarray Sum (5) Sudoku (5) Threaded (5) Word Search (5) jiuzhang (5) 1point3acres (4) Abbreviation (4) Anagram (4) Anagrams (4) Approximate Algorithm (4) Backtracking-Include vs Exclude (4) Brute Force - Enumeration (4) Chess Game (4) Consistent Hash (4) Dequeue (4) Distributed (4) Eulerian Cycle (4) Graph-Classic (4) Greedy (4) HackerRank AI (4) Histogram (4) Kadane Max Sum (4) Knapsack - Mixed (4) Knapsack - Unbounded (4) LeetCode - Recursive (4) MST (4) N Queens (4) Nerd Paradise (4) Parallel Algorithm (4) Practical Algorithm (4) Probability (4) Spell Checker (4) Stock Maximize (4) Subset Sum (4) Subsets (4) Symbol Table (4) Triangle (4) Water Jug (4) algnotes (4) fgdsb (4) to-do-2 (4) 最大化最小值 (4) 树形DP (4) 男人八题 (4) A Star (3) Algorithm Design (3) B Tree (3) Bellman Ford (3) Big Data Algorithm (3) Caterpillar Method (3) Coins (3) Company - Groupon (3) Company - Indeed (3) Cumulative Sum (3) DP-Fill by Length (3) DP-Two Variables (3) Dedup (3) Dropbox (3) Easy (3) Factor (3) Finite Automata (3) Github (3) GoLang (3) Graph - Bipartite (3) Graph Coloring (3) Include vs Exclude (3) Islands (3) Joseph (3) K (3) Knapsack-多重背包 (3) Knight (3) LeetCode - Bit (3) Linked List Merge Sort (3) LogN (3) Master Theorem (3) Median (3) Min Cost Flow (3) Minesweeper (3) Missing Numbers (3) NP Hard (3) Online Algorithm (3) Parentheses (3) Parser (3) Pascal's Triangle (3) Pattern Match (3) Programcreek (3) Project Euler (3) Rectangle (3) Robot (3) Scala (3) SegmentFault (3) Shuffle (3) Sieve of Eratosthenes (3) Skyline (3) Stack - Smart (3) State Machine (3) Strongly Connected Components (3) Subtree (3) TSP (3) Transform Tree (3) Trie + DFS (3) Warshall Floyd (3) With Random Pointer (3) Word Ladder (3) bookkeeping (3) codebytes (3) Activity Selection Problem (2) Advanced Algorithm (2) AnAlgorithmADay (2) Application of Algorithm (2) Array Merge (2) Augmented BST (2) BOJ (2) BT - Path Sum (2) Balanced Binary Search Tree (2) Binary Search - Smart (2) Binomial Coefficient (2) Bit Counting (2) Bit-Difficult (2) Bloom Filter (2) Book Coding Interview (2) Branch and Bound Method (2) Cache (2) Clock (2) Codesays (2) Company - Baidu (2) Company-Snapchat (2) Complete Binary Tree (2) DP - DFS (2) DP - Trie (2) DP-3D Table (2) DP-Classical (2) DP-Output Solution (2) DP-Slide Window Gap (2) DP-i-k-j (2) Distributed Algorithms (2) Divide and Conqure (2) Doubly Linked List (2) Edit Distance (2) Forward && Backward Scan (2) Game (2) GoHired (2) Graham Scan (2) Graph BFS+DFS (2) Graph-Cut Vertices (2) Hamiltonian Cycle (2) Hard Algorithm (2) Huffman Tree (2) In-order Traverse (2) Include or Exclude Last Element (2) Information Retrieval (2) Interview - Linkedin (2) Invariant (2) Jump Game (2) LeetCode - Hard (2) Linked Interview (2) Linked List Sort (2) Longest SubArray (2) Lucene-Solr (2) Math-Remainder Queue (2) Matrix Power (2) Minimum Vertex Cover (2) Negative All Values (2) Number Each Digit (2) Numerical Method (2) O(N) Hard (2) Object Design (2) Order Statistic Tree (2) Parent-Only Tree (2) Peak (2) PreProcess (2) Programming (2) Range Minimum Query (2) Regular Expression (2) Return Multiple Values (2) Reuse Forward Backward (2) Reverse (2) Rosettacode (2) Scan from right (2) Search (2) SimHash (2) Simple Algorithm (2) Sparse Table (2) Spatial Index (2) TV (2) Tile (2) Traversal From End (2) Tree Sum (2) Tree Traversal Return Multiple Values (2) Tree Without Tree Predefined (2) Two Pointers Window (2) Two-End-BFS (2) Word Break (2) Word Graph (2) Word Trie (2) Yahoo Interview (2) Young Tableau (2) 剑指Offer (2) 数位DP (2) 1-X (1) 51Nod (1) ? (1) Akka (1) Algorithm - New (1) Algorithm Series (1) Algorithms Part I (1) All Substrings (1) Analysis of Algorithm (1) Array-Element Index Negative (1) Array-Rearrange (1) Augmented Tree (1) Auxiliary Array (1) Auxiliary Array: Inc&Dec (1) BACK (1) BFS - Priority Queue (1) BFS Hard (1) BK-Tree (1) BZOJ (1) Basic (1) Bayes (1) Beauty of Math (1) Big Integer (1) Big Number (1) Binary (1) Binary String (1) Binary Tree Variant (1) Bipartite (1) Bit-Missing Number (1) BitMap (1) BitMap index (1) BitSet (1) Bug Free Code (1) BuildIt (1) C/C++ (1) CC Interview (1) Calculate Height at Same Recusrion (1) Cartesian tree (1) Check Tree Property (1) Chinese (1) Circular (1) Circular Buffer (1) Clean Code (1) Cloest (1) Clone (1) Code Quality (1) Codesolutiony (1) Company - Alibaba (1) Company - Palantir (1) Company - WalmartLabs (1) Company-Apple (1) Company-Epic (1) Company-Salesforce (1) Company-Yelp (1) Compression Algorithm (1) Concise (1) Concurrency (1) Convert BST to DLL (1) Convert DLL to BST (1) Custom Sort (1) Cyclic Replacement (1) DFS + RL (1) DFS+BFS, Flood Fill (1) DFS-Matrix (1) DI (1) DP Fill Diagonal First (1) DP-Difficult (1) DP-End with 0 or 1 (1) DP-Fill Diagonal First (1) DP-Graph (1) DP-Left and Right Array (1) DP-MaxMin (1) DP-Memoization (1) DP-Node All Possibilities (1) DP-Optimization (1) DP-Preserve Previous Value (1) DP-Print All Solution (1) DP-树形 (1) Database (1) Detect Negative Cycle (1) Diagonal (1) Directed Graph (1) Do Two Things at Same Recusrion (1) Domino (1) Dr Dobb's (1) Duplicate (1) EntryPoint (1) Equal probability (1) External Sort (1) FST (1) Failure Function (1) Fraction (1) Front End Pointers (1) Funny (1) Fuzzy String Search (1) Generating Function (1) Generation (1) Genetic algorithm (1) GeoHash (1) Geometry - Orientation (1) Google APAC (1) Graph But No Graph (1) Graph Transpose (1) Graph Traversal (1) Graph-Coloring (1) Graph-Longest Path (1) Gray Code (1) HOJ (1) Hanoi (1) How Hash (1) How to Test (1) Improve It (1) In Place (1) Inorder-Reverse Inorder Traverse Simultaneously (1) Interpolation search (1) Interview (1) Interview - Facebook (1) Isomorphic (1) JDK8 (1) K Dimensional Tree (1) Knapsack - Fractional (1) Knapsack - ZeroOnePack (1) Knuth Shuffle (1) Kosaraju’s algorithm (1) Kruskal (1) Kth Element (1) Lazy (1) Least Common Ancestor (1) LeetCode - Binary Tree (1) LeetCode - Coding (1) LeetCode - Construction (1) LeetCode - Detail (1) LeetCode - Related (1) LeetCode -P (1) Linked List Reverse (1) Linkedin (1) Linkedin Interview (1) Local MinMax (1) Lock (1) Logic Pattern (1) Longest Common Subsequence (1) Longest Common Substring (1) Longest Prefix Suffix(LPS) (1) Machine Learning (1) Maintain State (1) Manhattan Distance (1) Map && Reverse Map (1) Math - Induction (1) Math-Multiply (1) Math-Sum Of Digits (1) Matrix - O(N+M) (1) Matrix BFS (1) Matrix Exponentiation (1) Matrix Graph (1) Matrix Multiplication (1) Matrix Search (1) Matrix+DP (1) Matrix-Rotate (1) Max Min So Far (1) Memory-Efficient (1) MinHash (1) MinMax Heap (1) Monto Carlo (1) Multi-End BFS (1) Multi-Reverse (1) Multiple DFS (1) Multiple Steps (1) Multiple Tasks (1) Next Element (1) Next Successor (1) O(32N) (1) Offline Algorithm (1) Optimal Play (1) Optimization (1) PAT (1) Parenthesis (1) Partition (1) Path Finding (1) Patience Sort (1) Persistent (1) Pigeon Hole Principle (1) Power Set (1) Pratical Algorithm (1) Probabilistic Data Structure (1) Probability DP (1) Python (1) Queue & Stack (1) RSA (1) Ranking (1) Rddles (1) ReHash (1) Realtime (1) Recurrence Relation (1) Recursion (1) Recursive DFS (1) Recursive to Iterative (1) Red-Black Tree (1) Region (1) Remap (1) Resources (1) Reverse Inorder Traversal (1) Robin (1) Selection (1) Self Balancing BST (1) Similarity (1) Simplify (1) Sort && Binary Search (1) Space Complexity (1) Square (1) Strategy (1) Streaming Algorithm (1) String Algorithm. Symbol Table (1) String DP (1) String Distance (1) SubMatrix (1) Subsequence (1) System of Difference Constraints(差分约束系统) (1) Ternary Search Tree (1) Test (1) Test Cases (1) Thread (1) TimSort (1) Top-Down (1) Tournament (1) Tournament Tree (1) Trade Off (1) Transform Tree in Place (1) Tree Diameter (1) Tree Rotate (1) Trie and Heap (1) Trie vs Hash (1) Trie vs HashMap (1) Triplet (1) Two Data Structures (1) Two Stacks (1) USACO - Classical (1) USACO - Problems (1) UyHiP (1) Valid Tree (1) Vector (1) Virtual Matrix (1) Wiggle Sort (1) Wikipedia (1) ZOJ (1) ZigZag (1) baozitraining (1) codevs (1) cos126 (1) cycl (1) javabeat (1) jum (1) namic Programming (1) sqrt(N) (1) 两次dijkstra (1) 九度 (1) 二进制枚举 (1) 夹逼法 (1) 归一化 (1) 折半枚举 (1) 枚举 (1) 状态压缩DP (1) 英雄会 (1) 逆向思维 (1)

Popular Posts