Sunday, July 3, 2016

Missing Int


/* 10.3 Given an input file with four billion non-negative integers, provide an
 * algorithm to generate an integer which is not contained in the file. Assume
 * you have 1 GB of memory available for this task. FOLLOW UP What if you have
 * only 10 M8 of memory? Assume that all the values are distinct and we now have
 * no more than one billion non-negative integers.

 * There are a total of 2^32, or 4 billion, distinct integers possible(and 2^31
 * non-negative integers). We have 1 GB of memory, or 8 billion bits.
 * 
 * Thus, with 8 billion bits, we can map all possible integers to a distinct bit
 * with the available memory. The logic is as follows: 1, Create a bit
 * vector(BV) with 4 billion bits. Recal that a bit vector is an array that
 * compactly stores between values by using an array of ints(or another data
 * type) Each int stores a sequence of 32 bits, or boolean values. 2, Initialize
 * BV with all 0's. 3, Scan all numbers(num) from the file and call
 * BV.set(num,1). 4, Now scan again BV from the 0th index. 5, Return the first
 * index which has a value of 0.
 */
public class GenerateInteger3 {
long numberOfInts = ((long) Integer.MAX_VALUE) + 1;
byte[] bitfield = new byte[(int) (numberOfInts / 8)];

void findOpenNumber() throws FileNotFoundException {
in = new Scanner(new FileReader("file.txt"));
while (in.hasNextInt()) {
int n = in.nextInt();
/*
* finds the corrsponding number in the bitfield by using the OR
* operator to set the nth bit of a byte(e.g. , 10 would correspond
* to the 2nd bit of index 2 in the byte array
*/

bitfield[n / 8] |= 1 << (n % 8);
}

for (int i = 0; i < bitfield.length; i++) {
for (int j = 0; j < 8; j++) {
/*
* retrieves the individual bits of eachbyte. when 0 bit is
* found, finds the correspondingvalue
*/

if ((bitfield[i] & (1 << j)) == 0) {
System.out.println(i * 8 + j);
extracted();
return;
}
}
}
}

/*
* Follow Up: what if we have only 10 MB memory? It's possible to find a
* missing integer with two passes of the data set. We can divide up the
* integers into blocks of some size(we will discuss how to decide on a size
* later). Let's just assume that we divide up the integers into blocks of
* 1000. So, block 0 represents the numbers 0 through 999, block 1
* represents numbers 1000-1999, and so on.
* Since all the values are distinct, we know how many values we should find
* in each block. So, we search through the file and count how many values
* are between 0 and 999, how many are between 1000 and 1999, and so on. If
* we count only 999 values in a particular range, then we know that a
* missing int must be in that range.
* In the second pass, we will actually look for which number in that range
* is missing. We use the bit vector approach from the first part of this
* problem. We can ignore any number outside of this specific range.
* The question, now is what is the appropriate block size? Let's define
* some variables as follows: Let rangeSize be the size of the ranges that
* each block in the first pass represents. Let arraySize represent the
* number of blocks in the first pass. Note that arraySize=2^31/rangeSize,
* since there are 2^31 non-negative integers.
* We need to select a value for rangeSize such that the memory from the
* first pass(the array) and the second pass(the bit vector) fit.
*/

/*
* First pass: The array The array in the first pass can fit in 10
* megabytes, or roughly 2^23 bytes, of memory. Since each element in the
* array is an int, and an int is 4 bytes, we can hold an array of at most
* about 2^21 elements. So, we can deduce the following:
* arraySize=2^31/rangeSize<=s^21 rangeSize>=2^31/2^21 rangeSize>=2^10
*/

/*
* Second Pass: The Bit Vector We need to have enough space to store
* rangeSize bits. Since we can fit 2^23 bytes in memory, we can fit 2^26
* bits in memory. Therefore, we can conclude the following: 2^10
* rangeSize<=2^26 These conditions give us a good amount of wiggle room,
* but hte nearer to the middle that we pick, the less memory will be used
* at any given time.
*/
int bitsize = 1048576;// 2^20 bits(2^17 bytes)
int blockNum = 4096;// 2^12
byte[] bitfield1 = new byte[bitsize / 8];

int[] blocks = new int[blockNum];
private Scanner in;

@SuppressWarnings("resource")
void findOpenNumber1() throws FileNotFoundException {
int starting = -1;
Scanner in = new Scanner(new FileReader("file.txt"));
while (in.hasNextInt()) {
int n = in.nextInt();
blocks[n / (bitfield1.length * 8)]++;
}

for (int i = 0; i < blocks.length; i++) {
if (blocks[i] < bitfield1.length * 8) {
// if value < 2^20, then at least 1 number if missing
// in that section
starting = i * bitfield1.length * 8;
break;
}
}
in = new Scanner(new FileReader("file.txt"));
while (in.hasNextInt()) {
int n = in.nextInt();
// if the number if inside the block that is missing
// numbers, we record it
if (n >= starting && n < starting + bitfield.length * 8) {
bitfield[(n - starting) / 8] |= 1 << ((n - starting) % 8);
}
}

for (int i = 0; i < bitfield.length; i++) {
for (int j = 0; j < 8; j++) {
// retrieves the individual bits of each byte,
// when 0 bit is found, finds the
// corresponding value
if ((bitfield[i] & (1 << j)) == 0) {
System.out.println(i * 8 + j + starting);
//extracted();
return;
}
}
}
}

private void extracted() {
return;
}

}

/*
 * What if, as a potential follow up question, the interviewer asked you to
 * solve the problem with even less memory? In this case we would do repeated
 * passes using the approach from the first step. We would first check to see
 * how many integers are found within each sequence of a million elements. Then,
 * in the second pass, we would check how many integers are found in each
 * sequence of a thousand elements. Finally, in the third pass, we would apply
 * the bit vector.
 */

int findOpenNumber(String filename) throws FileNotFoundException {
        int rangeSize = (1 << 20); // 2 A20 bits (2A17 bytes)
        /* Get count of number of values within each block. */
        int[] blocks = getCountPerBlock(filename, rangeSize);
        /* Find a block with a missing value. */
        int blocklndex = findBlockWithMissing(blocks, rangeSize);
        if (blocklndex < 0) return -1;

        /* Create bit vector for items within this range. */
        byte[] bitVector = getBitVectorForRange(filename, blockindex, rangeSize);

        /* Find a zero in the bit vector */
        int offset findZero(bitVector);
        if (offset < 0) return -1;

        /* Compute missing value. */
        return blockindex * rangeSize + offset;
}

/* Get count of items within each range. */
int[] getCountPerBlock(String filename, int rangeSize) throws FileNotFoundException {
        int arraySize = Integer.MAX_VALUE / rangeSize + 1;
        int[] blocks = new int[arraySize];

        Scanner in = new Scanner (new FileReader(filename));
        while (in.hasNextint()) {
                int value = in.nextint();
                blocks[value / rangeSize]++;
        }
        in.close();
        return blocks;
}

/* Find a block whose count is low. */
int findBlockWithMissing(int[] blocks, int rangeSize) {
        for (int i= 0; i < blocks.length; i++) {
                if (blocks[i] < rangeSize) {
                        return i;
                }
        }
        return -1;
}

/* Create a bit vector for the values within a specific range. */
byte[] getBitVectorForRange(String filename, int blockindex, int rangeSize)
throws FileNotFoundException {
  int startRange = blockindex * rangeSize;
  int endRange = startRange + rangeSize;
  byte[] bitVector = new byte[rangeSize/Byte.SIZE];

  Scanner in = new Scanner(new FileReader(filename));
  while (in.hasNextint()) {
    int value = in.nextint();
    /* If the number is inside the block that's missing numbers, we record it */
    if (startRange <= value && value < endRange) {
      int offset = value - startRange;
      int mask = (1 <<(offset% Byte.SIZE));
      bitVector[offset / Byte.SIZE) |= mask;
    }
  }
  in.close();
  return bitVector;

}

/* Find bit index that is 0 within byte. */
int findZero(byte b) {
        for (int i= 0; i < Byte.SIZE; i++) {
                int mask= 1 << i;
                if ((b & mask)== 0) {
                        return i;
                }
        }
        return -1;
}

/* Find a zero within the bit vector and return the index. */
int findZero(byte[] bitVector) {
        for (int i= 0; i < bitVector.length; i++) {
                if (bitVector[i] != -0) {//If not all ls
                        int bitindex = findZero(bitVector[i]);
                        return i *Byte.SIZE+ bitindex;
                }
        }
        return -1;
}

No comments:

Post a Comment

Labels

GeeksforGeeks (1107) LeetCode (985) Algorithm (795) Review (759) to-do (631) LeetCode - Review (506) Classic Algorithm (324) Dynamic Programming (292) Classic Interview (288) Google Interview (242) Tree (145) POJ (139) Difficult Algorithm (132) LeetCode - Phone (127) EPI (125) Different Solutions (120) Bit Algorithms (118) Lintcode (113) Cracking Coding Interview (110) Smart Algorithm (109) Math (107) HackerRank (89) Binary Search (81) Binary Tree (80) Graph Algorithm (74) Greedy Algorithm (72) DFS (66) LeetCode - Extended (62) Interview Corner (61) Stack (60) List (58) Advanced Data Structure (56) Codility (54) BFS (53) ComProGuide (52) Algorithm Interview (50) Geometry Algorithm (48) Binary Search Tree (46) USACO (46) Trie (45) Mathematical Algorithm (42) ACM-ICPC (41) Interval (41) Data Structure (40) Knapsack (40) Space Optimization (40) Jobdu (39) Recursive Algorithm (39) LeetCode Hard (38) Matrix (38) String Algorithm (38) Backtracking (36) Codeforces (36) Introduction to Algorithms (36) Must Known (36) Beauty of Programming (35) Sort (35) Union-Find (34) Array (33) prismoskills (33) Segment Tree (32) Sliding Window (32) Data Structure Design (31) HDU (31) Google Code Jam (30) Permutation (30) Puzzles (30) Array O(N) (29) Company-Airbnb (29) Company-Zenefits (28) Microsoft 100 - July (28) Palindrome (28) to-do-must (28) Priority Queue (27) Random (27) Graph (26) Company - LinkedIn (25) GeeksQuiz (25) Logic Thinking (25) Pre-Sort (25) hihocoder (25) Queue (24) Company-Facebook (23) High Frequency (23) TopCoder (23) Algorithm Game (22) Hash (22) Post-Order Traverse (22) Binary Indexed Trees (21) Bisection Method (21) DFS + Review (21) Lintcode - Review (21) Brain Teaser (20) CareerCup (20) Company - Twitter (20) Merge Sort (20) Follow Up (19) O(N) (19) Time Complexity (19) Two Pointers (19) UVA (19) Ordered Stack (18) Probabilities (18) Company-Uber (17) Game Theory (17) Topological Sort (17) Codercareer (16) Heap (16) Shortest Path (16) String Search (16) Tree Traversal (16) itint5 (16) Difficult (15) Iterator (15) BST (14) Number (14) Number Theory (14) Amazon Interview (13) Basic Algorithm (13) Codechef (13) Euclidean GCD (13) KMP (13) Long Increasing Sequence(LIS) (13) Majority (13) mitbbs (13) Combination (12) Computational Geometry (12) LeetCode - Classic (12) Modify Tree (12) Reconstruct Tree (12) Reservoir Sampling (12) Reverse Thinking (12) 尺取法 (12) AOJ (11) DFS+Backtracking (11) Fast Power Algorithm (11) Graph DFS (11) LCA (11) LeetCode - DFS (11) Miscs (11) Princeton (11) Proof (11) Tree DP (11) X Sum (11) 挑战程序设计竞赛 (11) Bisection (10) Bucket Sort (10) Coin Change (10) Company - Microsoft (10) DFS+Cache (10) Facebook Hacker Cup (10) HackerRank Easy (10) O(1) Space (10) Rolling Hash (10) SPOJ (10) Theory (10) Tutorialhorizon (10) DP-Multiple Relation (9) DP-Space Optimization (9) Divide and Conquer (9) Kadane - Extended (9) Mathblog (9) Max-Min Flow (9) Prefix Sum (9) Quick Sort (9) Simulation (9) Stack Overflow (9) Stock (9) System Design (9) TreeMap (9) Use XOR (9) Book Notes (8) Bottom-Up (8) Company-Amazon (8) DFS+BFS (8) LeetCode - DP (8) Left and Right Array (8) Linked List (8) Longest Common Subsequence(LCS) (8) Prime (8) Suffix Tree (8) Tech-Queries (8) Traversal Once (8) 穷竭搜索 (8) Algorithm Problem List (7) Expression (7) Facebook Interview (7) Fibonacci Numbers (7) Game Nim (7) Graph BFS (7) HackerRank Difficult (7) Hackerearth (7) Interval Tree (7) Inversion (7) Kadane’s Algorithm (7) Level Order Traversal (7) Math-Divisible (7) Probability DP (7) Quick Select (7) Radix Sort (7) n00tc0d3r (7) 蓝桥杯 (7) Catalan Number (6) Classic Data Structure Impl (6) DFS+DP (6) DP - Tree (6) DP-Print Solution (6) Dijkstra (6) Dutch Flag (6) How To (6) Interviewstreet (6) Knapsack - MultiplePack (6) Manacher (6) Minimum Spanning Tree (6) Morris Traversal (6) Multiple Data Structures (6) One Pass (6) Programming Pearls (6) Pruning (6) Rabin-Karp (6) Randomized Algorithms (6) Sampling (6) Schedule (6) Stream (6) Suffix Array (6) Threaded (6) TreeSet (6) Xpost (6) reddit (6) AI (5) Algorithm - Brain Teaser (5) Art Of Programming-July (5) Big Data (5) Brute Force (5) Code Kata (5) Codility-lessons (5) Coding (5) Company - WMware (5) Convex Hull (5) Crazyforcode (5) Cycle (5) DP-Include vs Exclude (5) Fast Slow Pointers (5) Graph Cycle (5) Hash Strategy (5) Immutability (5) Java (5) Matrix Chain Multiplication (5) Maze (5) Microsoft Interview (5) Pre-Sum (5) Quadtrees (5) Quick Partition (5) Quora (5) SPFA(Shortest Path Faster Algorithm) (5) Subarray Sum (5) Sudoku (5) Sweep Line (5) Word Search (5) jiuzhang (5) 单调栈 (5) 树形DP (5) 1point3acres (4) Abbreviation (4) Anagram (4) Anagrams (4) Approximate Algorithm (4) Backtracking-Include vs Exclude (4) Brute Force - Enumeration (4) Chess Game (4) Consistent Hash (4) Distributed (4) Eulerian Cycle (4) Find Rule (4) Flood fill (4) Graph-Classic (4) HackerRank AI (4) Histogram (4) Kadane Max Sum (4) Knapsack - Mixed (4) Knapsack - Unbounded (4) LeetCode - Recursive (4) LeetCode - TODO (4) MST (4) MinMax (4) N Queens (4) Nerd Paradise (4) Parallel Algorithm (4) Practical Algorithm (4) Probability (4) Programcreek (4) Spell Checker (4) Stock Maximize (4) Subset Sum (4) Subsets (4) Symbol Table (4) Triangle (4) Water Jug (4) algnotes (4) fgdsb (4) to-do-2 (4) 最大化最小值 (4) A Star (3) Algorithm - How To (3) Algorithm Design (3) B Tree (3) Big Data Algorithm (3) Caterpillar Method (3) Coins (3) Company - Groupon (3) Company - Indeed (3) Cumulative Sum (3) DP-Fill by Length (3) DP-Two Variables (3) Dedup (3) Dequeue (3) Dropbox (3) Easy (3) Finite Automata (3) Github (3) GoLang (3) Graph - Bipartite (3) Include vs Exclude (3) Joseph (3) Jump Game (3) K (3) Knapsack-多重背包 (3) LeetCode - Bit (3) Linked List Merge Sort (3) LogN (3) Master Theorem (3) Min Cost Flow (3) Minesweeper (3) Missing Numbers (3) NP Hard (3) O(N) Hard (3) Online Algorithm (3) Pascal's Triangle (3) Pattern Match (3) Project Euler (3) Rectangle (3) Scala (3) SegmentFault (3) Shuffle (3) Sieve of Eratosthenes (3) Stack - Smart (3) State Machine (3) Subtree (3) Transform Tree (3) Trie + DFS (3) Two Pointers Window (3) Warshall Floyd (3) With Random Pointer (3) Word Ladder (3) bookkeeping (3) codebytes (3) Activity Selection Problem (2) Advanced Algorithm (2) AnAlgorithmADay (2) Application of Algorithm (2) Array Merge (2) BOJ (2) BT - Path Sum (2) Balanced Binary Search Tree (2) Bellman Ford (2) Binary Search - Smart (2) Binomial Coefficient (2) Bit Counting (2) Bit Mask (2) Bit-Difficult (2) Bloom Filter (2) Book Coding Interview (2) Branch and Bound Method (2) Clock (2) Codesays (2) Company - Baidu (2) Company-Snapchat (2) Complete Binary Tree (2) DFS+BFS, Flood Fill (2) DP - DFS (2) DP-3D Table (2) DP-Classical (2) DP-Output Solution (2) DP-Slide Window Gap (2) DP-i-k-j (2) DP-树形 (2) Distributed Algorithms (2) Divide and Conqure (2) Doubly Linked List (2) Edit Distance (2) Factor (2) Forward && Backward Scan (2) GoHired (2) Graham Scan (2) Graph BFS+DFS (2) Graph Coloring (2) Graph-Cut Vertices (2) Hamiltonian Cycle (2) Huffman Tree (2) In-order Traverse (2) Include or Exclude Last Element (2) Information Retrieval (2) Interview - Linkedin (2) Invariant (2) Islands (2) Linked Interview (2) Linked List Sort (2) Longest SubArray (2) Lucene-Solr (2) Math-Remainder Queue (2) Matrix Power (2) Median (2) Minimum Vertex Cover (2) Negative All Values (2) Number Each Digit (2) Numerical Method (2) Object Design (2) Order Statistic Tree (2) Parent-Only Tree (2) Parentheses (2) Parser (2) Peak (2) Programming (2) Range Minimum Query (2) Regular Expression (2) Return Multiple Values (2) Reuse Forward Backward (2) Robot (2) Rosettacode (2) Scan from right (2) Search (2) SimHash (2) Simple Algorithm (2) Skyline (2) Spatial Index (2) Strongly Connected Components (2) Summary (2) TV (2) Tile (2) Traversal From End (2) Tree Sum (2) Tree Traversal Return Multiple Values (2) Word Break (2) Word Graph (2) Word Trie (2) Yahoo Interview (2) Young Tableau (2) 剑指Offer (2) 数位DP (2) 1-X (1) 51Nod (1) Akka (1) Algorithm - New (1) Algorithm Series (1) Algorithms Part I (1) Analysis of Algorithm (1) Array-Element Index Negative (1) Array-Rearrange (1) Augmented BST (1) Auxiliary Array (1) Auxiliary Array: Inc&Dec (1) BACK (1) BK-Tree (1) BZOJ (1) Basic (1) Bayes (1) Beauty of Math (1) Big Integer (1) Big Number (1) Binary (1) Binary Sarch Tree (1) Binary String (1) Binary Tree Variant (1) Bipartite (1) Bit-Missing Number (1) BitMap (1) BitMap index (1) BitSet (1) Bug Free Code (1) BuildIt (1) C/C++ (1) CC Interview (1) Cache (1) Calculate Height at Same Recusrion (1) Cartesian tree (1) Check Tree Property (1) Chinese (1) Circular Buffer (1) Cloest (1) Clone (1) Code Quality (1) Codesolutiony (1) Company - Alibaba (1) Company - Palantir (1) Company - WalmartLabs (1) Company-Apple (1) Company-Epic (1) Company-Salesforce (1) Company-Yelp (1) Compression Algorithm (1) Concurrency (1) Cont Improvement (1) Convert BST to DLL (1) Convert DLL to BST (1) Custom Sort (1) Cyclic Replacement (1) DFS-Matrix (1) DP - Probability (1) DP Fill Diagonal First (1) DP-Difficult (1) DP-End with 0 or 1 (1) DP-Fill Diagonal First (1) DP-Graph (1) DP-Left and Right Array (1) DP-MaxMin (1) DP-Memoization (1) DP-Node All Possibilities (1) DP-Optimization (1) DP-Preserve Previous Value (1) DP-Print All Solution (1) Database (1) Detect Negative Cycle (1) Diagonal (1) Directed Graph (1) Do Two Things at Same Recusrion (1) Domino (1) Dr Dobb's (1) Duplicate (1) Equal probability (1) External Sort (1) FST (1) Failure Function (1) Fraction (1) Front End Pointers (1) Funny (1) Fuzzy String Search (1) Game (1) Generating Function (1) Generation (1) Genetic algorithm (1) GeoHash (1) Geometry - Orientation (1) Google APAC (1) Graph But No Graph (1) Graph Transpose (1) Graph Traversal (1) Graph-Coloring (1) Graph-Longest Path (1) Gray Code (1) HOJ (1) Hanoi (1) Hard Algorithm (1) How Hash (1) How to Test (1) Improve It (1) In Place (1) Inorder-Reverse Inorder Traverse Simultaneously (1) Interpolation search (1) Interview (1) Interview - Facebook (1) Isomorphic (1) JDK8 (1) K Dimensional Tree (1) Knapsack - Fractional (1) Knapsack - ZeroOnePack (1) Knight (1) Knuth Shuffle (1) Kosaraju’s algorithm (1) Kruskal (1) Kth Element (1) Least Common Ancestor (1) LeetCode - Binary Tree (1) LeetCode - Coding (1) LeetCode - Detail (1) LeetCode - Related (1) Linked List Reverse (1) Linkedin (1) Linkedin Interview (1) Local MinMax (1) Logic Pattern (1) Longest Common Subsequence (1) Longest Common Substring (1) Longest Prefix Suffix(LPS) (1) Machine Learning (1) Maintain State (1) Manhattan Distance (1) Map && Reverse Map (1) Math - Induction (1) Math-Multiply (1) Math-Sum Of Digits (1) Matrix - O(N+M) (1) Matrix BFS (1) Matrix Graph (1) Matrix Search (1) Matrix+DP (1) Matrix-Rotate (1) Max Min So Far (1) Memory-Efficient (1) MinHash (1) MinMax Heap (1) Monotone Queue (1) Monto Carlo (1) Multi-End BFS (1) Multi-Reverse (1) Multiple DFS (1) Multiple Tasks (1) Next Element (1) Next Successor (1) Offline Algorithm (1) PAT (1) Parenthesis (1) Partition (1) Path Finding (1) Patience Sort (1) Persistent (1) Pigeon Hole Principle (1) Power Set (1) Pratical Algorithm (1) PreProcess (1) Probabilistic Data Structure (1) Python (1) Queue & Stack (1) RSA (1) Ranking (1) Rddles (1) ReHash (1) Realtime (1) Recurrence Relation (1) Recursive DFS (1) Recursive to Iterative (1) Red-Black Tree (1) Region (1) Resources (1) Reverse Inorder Traversal (1) Robin (1) Selection (1) Self Balancing BST (1) Similarity (1) Sort && Binary Search (1) Square (1) Streaming Algorithm (1) String Algorithm. Symbol Table (1) String DP (1) String Distance (1) SubMatrix (1) Subsequence (1) System of Difference Constraints(差分约束系统) (1) TSP (1) Ternary Search Tree (1) Test (1) Test Cases (1) Thread (1) TimSort (1) Top-Down (1) Tournament (1) Tournament Tree (1) Transform Tree in Place (1) Tree Diameter (1) Tree Rotate (1) Tree Without Tree Predefined (1) Trie and Heap (1) Trie vs Hash (1) Trie vs HashMap (1) Triplet (1) Two Data Structures (1) Two Stacks (1) USACO - Classical (1) USACO - Problems (1) UyHiP (1) Valid Tree (1) Vector (1) Virtual Matrix (1) Wiggle Sort (1) Wikipedia (1) ZOJ (1) ZigZag (1) baozitraining (1) codevs (1) cos126 (1) javabeat (1) jum (1) namic Programming (1) sqrt(N) (1) 两次dijkstra (1) 九度 (1) 二进制枚举 (1) 夹逼法 (1) 归一化 (1) 折半枚举 (1) 枚举 (1) 状态压缩DP (1) 男人八题 (1) 英雄会 (1) 逆向思维 (1)

Popular Posts