http://coursera.cs.princeton.edu/algs4/assignments/wordnet.html
WordNet is a semantic lexicon for the English language that is used extensively by computational linguists and cognitive scientists; for example, it was a key component in IBM's Watson. WordNet groups words into sets of synonyms called synsets and describes semantic relationships between them. One such relationship is the is-a relationship, which connects a hyponym (more specific synset) to a hypernym (more general synset). For example, animal is a hypernym of both bird and fish; bird is a hypernym of eagle, pigeon, and seagull.
The WordNet digraph. Your first task is to build the wordnet digraph: each vertex v is an integer that represents a synset, and each directed edge v→w represents that w is a hypernym of v. The wordnet digraph is a rooted DAG: it is acyclic and has one vertex—the root—that is an ancestor of every other vertex. However, it is not necessarily a tree because a synset can have more than one hypernym. A small subgraph of the wordnet digraph is illustrated below.
The WordNet input file formats. We now describe the two data files that you will use to create the wordnet digraph. The files are in CSV format: each line contains a sequence of fields, separated by commas.
Performance requirements. Your data type should use space linear in the input size (size of synsets and hypernyms files). The constructor should take time linearithmic (or better) in the input size. The method isNoun()should run in time logarithmic (or better) in the number of nouns. The methods distance() and sap() should run in time linear in the size of the WordNet digraph. For the analysis, assume that the number of nouns per synset is bounded by a constant.
Shortest ancestral path. An ancestral path between two vertices v and w in a digraph is a directed path from v to a common ancestor x, together with a directed path from w to the same ancestor x. A shortest ancestral path is an ancestral path of minimum total length. For example, in the digraph at left (digraph1.txt), the shortest ancestral path between 3 and 11 has length 4 (with common ancestor 1). In the digraph at right (digraph2.txt), one ancestral path between 1 and 5 has length 4 (with common ancestor 5), but the shortest ancestral path has length 2 (with common ancestor 0).
SAP data type. Implement an immutable data type SAP with the following API:
Performance requirements. All methods (and the constructor) should take time at most proportional to E + V in the worst case, where E and V are the number of edges and vertices in the digraph, respectively. Your data type should use space proportional to E + V.
Test client. The following test client takes the name of a digraph input file as as a command-line argument, constructs the digraph, reads in vertex pairs from standard input, and prints out the length of the shortest ancestral path between the two vertices and a common ancestor that participates in that path:
We define the semantic relatedness of two wordnet nouns A and B as follows:
Outcast detection. Given a list of wordnet nouns A1, A2, ..., An, which noun is the least related to the others? To identify an outcast, compute the sum of the distances between each noun and every other one:
Implement an immutable data type Outcast with the following API:
The following test client takes from the command line the name of a synset file, the name of a hypernym file, followed by the names of outcast files, and prints out an outcast in each file:
https://segmentfault.com/a/1190000005345079
WordNet is a semantic lexicon for the English language that is used extensively by computational linguists and cognitive scientists; for example, it was a key component in IBM's Watson. WordNet groups words into sets of synonyms called synsets and describes semantic relationships between them. One such relationship is the is-a relationship, which connects a hyponym (more specific synset) to a hypernym (more general synset). For example, animal is a hypernym of both bird and fish; bird is a hypernym of eagle, pigeon, and seagull.
The WordNet digraph. Your first task is to build the wordnet digraph: each vertex v is an integer that represents a synset, and each directed edge v→w represents that w is a hypernym of v. The wordnet digraph is a rooted DAG: it is acyclic and has one vertex—the root—that is an ancestor of every other vertex. However, it is not necessarily a tree because a synset can have more than one hypernym. A small subgraph of the wordnet digraph is illustrated below.
- List of noun synsets. The file synsets.txt lists all the (noun) synsets in WordNet. The first field is the synset id (an integer), the second field is the synonym set (or synset), and the third field is its dictionary definition (or gloss). For example, the line
36,AND_circuit AND_gate,a circuit in a computer that fires only when all of its inputs fire
- List of hypernyms. The file hypernyms.txt contains the hypernym relationships: The first field is a synset id; subsequent fields are the id numbers of the synset's hypernyms. For example, the following line
164,21012,56099
164,Actifed,trade name for a drug containing an antihistamine and a decongestant... 21012,antihistamine,a medicine used to treat allergies... 56099,nasal_decongestant,a decongestant that provides temporary relief of nasal...
Corner cases. All methods and the constructor should throw a java.lang.NullPointerException if any argument is null. The constructor should throw a java.lang.IllegalArgumentException if the input does not correspond to a rooted DAG. The distance() and sap() methods should throw a java.lang.IllegalArgumentException unless both of the noun arguments are WordNet nouns.public class WordNet { // constructor takes the name of the two input files public WordNet(String synsets, String hypernyms) // returns all WordNet nouns public Iterable<String> nouns() // is the word a WordNet noun? public boolean isNoun(String word) // distance between nounA and nounB (defined below) public int distance(String nounA, String nounB) // a synset (second field of synsets.txt) that is the common ancestor of nounA and nounB // in a shortest ancestral path (defined below) public String sap(String nounA, String nounB) // do unit testing of this class public static void main(String[] args) }
Performance requirements. Your data type should use space linear in the input size (size of synsets and hypernyms files). The constructor should take time linearithmic (or better) in the input size. The method isNoun()should run in time logarithmic (or better) in the number of nouns. The methods distance() and sap() should run in time linear in the size of the WordNet digraph. For the analysis, assume that the number of nouns per synset is bounded by a constant.
Shortest ancestral path. An ancestral path between two vertices v and w in a digraph is a directed path from v to a common ancestor x, together with a directed path from w to the same ancestor x. A shortest ancestral path is an ancestral path of minimum total length. For example, in the digraph at left (digraph1.txt), the shortest ancestral path between 3 and 11 has length 4 (with common ancestor 1). In the digraph at right (digraph2.txt), one ancestral path between 1 and 5 has length 4 (with common ancestor 5), but the shortest ancestral path has length 2 (with common ancestor 0).
SAP data type. Implement an immutable data type SAP with the following API:
Corner cases. All methods should throw a java.lang.NullPointerException if any argument is null. All methods should throw a java.lang.IndexOutOfBoundsException if any argument vertex is invalid—not between 0 and G.V() - 1.public class SAP { // constructor takes a digraph (not necessarily a DAG) public SAP(Digraph G) // length of shortest ancestral path between v and w; -1 if no such path public int length(int v, int w) // a common ancestor of v and w that participates in a shortest ancestral path; -1 if no such path public int ancestor(int v, int w) // length of shortest ancestral path between any vertex in v and any vertex in w; -1 if no such path public int length(Iterable<Integer> v, Iterable<Integer> w) // a common ancestor that participates in shortest ancestral path; -1 if no such path public int ancestor(Iterable<Integer> v, Iterable<Integer> w) // do unit testing of this class public static void main(String[] args) }
Performance requirements. All methods (and the constructor) should take time at most proportional to E + V in the worst case, where E and V are the number of edges and vertices in the digraph, respectively. Your data type should use space proportional to E + V.
Test client. The following test client takes the name of a digraph input file as as a command-line argument, constructs the digraph, reads in vertex pairs from standard input, and prints out the length of the shortest ancestral path between the two vertices and a common ancestor that participates in that path:
Here is a sample execution:public static void main(String[] args) { In in = new In(args[0]); Digraph G = new Digraph(in); SAP sap = new SAP(G); while (!StdIn.isEmpty()) { int v = StdIn.readInt(); int w = StdIn.readInt(); int length = sap.length(v, w); int ancestor = sap.ancestor(v, w); StdOut.printf("length = %d, ancestor = %d\n", length, ancestor); } }
Measuring the semantic relatedness of two nouns. Semantic relatedness refers to the degree to which two concepts are related. Measuring semantic relatedness is a challenging problem. For example, most of us agree that George Bush and John Kennedy (two U.S. presidents) are more related than are George Bush and chimpanzee (two primates). However, not most of us agree that George Bush and Eric Arthur Blair are related concepts. But if one is aware that George Bush and Eric Arthur Blair (aka George Orwell) are both communicators, then it becomes clear that the two concepts might be related.% more digraph1.txt % java SAP digraph1.txt 13 3 11 11 length = 4, ancestor = 1 7 3 8 3 9 12 3 1 length = 3, ancestor = 5 4 1 5 1 7 2 9 5 length = 4, ancestor = 0 10 5 11 10 1 6 12 10 length = -1, ancestor = -1 1 0 2 0
We define the semantic relatedness of two wordnet nouns A and B as follows:
- distance(A, B) = distance is the minimum length of any ancestral path between any synset v of A and any synset w of B.
Outcast detection. Given a list of wordnet nouns A1, A2, ..., An, which noun is the least related to the others? To identify an outcast, compute the sum of the distances between each noun and every other one:
di = dist(Ai, A1) + dist(Ai, A2) + ... + dist(Ai, An)and return a noun At for which dt is maximum.
Implement an immutable data type Outcast with the following API:
Assume that argument to outcast() contains only valid wordnet nouns (and that it contains at least two such nouns).public class Outcast { public Outcast(WordNet wordnet) // constructor takes a WordNet object public String outcast(String[] nouns) // given an array of WordNet nouns, return an outcast public static void main(String[] args) // see test client below }
The following test client takes from the command line the name of a synset file, the name of a hypernym file, followed by the names of outcast files, and prints out an outcast in each file:
http://coursera.cs.princeton.edu/algs4/checklists/wordnet.htmlpublic static void main(String[] args) { WordNet wordnet = new WordNet(args[0], args[1]); Outcast outcast = new Outcast(wordnet); for (int t = 2; t < args.length; t++) { In in = new In(args[t]); String[] nouns = in.readAllStrings(); StdOut.println(args[t] + ": " + outcast.outcast(nouns)); } }
https://segmentfault.com/a/1190000005345079
整个作业可以分为三大部分:选择合适的结构存储结构复杂的词典及其网络关系(基本功),实现正确高效的广度优先遍历计算SAP(难点),和在此基础上一层一层更细致的优化(进阶);
本次也是checklist优化部分唯一标注optional的作业,因为的确实现起来有难度;分布如上所注。
词典(synsets)为ID与单词的多对多关系(一词多号,一号多词,释义仅供参考无需读取),上下位关系(hypernyms)是以ID为单位的树形结构(图中的箭头关系),可以明显看出hypernyms使用有向图表示,synsets存储方式取决于访问需求,因程序中正反都会使用(以词求号,以号求词,求词存在),我选择了空间换时间:
本次也是checklist优化部分唯一标注optional的作业,因为的确实现起来有难度;分布如上所注。
词典(synsets)为ID与单词的多对多关系(一词多号,一号多词,释义仅供参考无需读取),上下位关系(hypernyms)是以ID为单位的树形结构(图中的箭头关系),可以明显看出hypernyms使用有向图表示,synsets存储方式取决于访问需求,因程序中正反都会使用(以词求号,以号求词,求词存在),我选择了空间换时间:
private ArrayList<LinkedList<String>> synset; // 以号求词(获取SAP中的ancestor对应的单词)
private HashMap<String, LinkedList<Integer>> ids; // 以词求号(获取索引单词的ID以计算SAP)
private HashSet<String> nouns; // 求词存在(API需求功能)
private SAP sap; // 有向图
文件读取过程稍繁琐,编程习惯要良好,保持谨慎不会错。
接着是SAP的实现,这里周旋的余地很大,一定要认真思考问题本质,并多考虑一些特殊情况检测自己的想法,再开始下手去做,并如checklist所强调,不要轻易尝试直接实现优化版本,因为细节繁琐不易考虑周全,而可能带来的潜在问题非常多,每实现一部分都要做周全的测试确保无误才行。(尽管最终成功的优化,也可能仅仅是几行的改变而已)
这里要提到的点(细节):
接着是SAP的实现,这里周旋的余地很大,一定要认真思考问题本质,并多考虑一些特殊情况检测自己的想法,再开始下手去做,并如checklist所强调,不要轻易尝试直接实现优化版本,因为细节繁琐不易考虑周全,而可能带来的潜在问题非常多,每实现一部分都要做周全的测试确保无误才行。(尽管最终成功的优化,也可能仅仅是几行的改变而已)
这里要提到的点(细节):
- 在checklist提到的三个优化选择中,我选择了相对保守,但工作良好的优化方式:每次的搜索初始化使用HashMap构造函数;正确实现了两端同步运行的BFS,并提供了提前结束的出口;只缓存了单个ID(单个节点与单个节点)的查询结果;
- 对单ID缓存的存储方式,我的symbol table使用了一种很直接的无重合又简单的hash计策:给定查询ID为v与w,总ID数为V,则查询结果的编号记为:
(V-1)+(V-2)+(V-3)+...+v+(w-v),化简后为(V-1)v-(vv+v)/2+w。 - 提前结束BFS的出口开始时没有考虑清楚,就暂时没有添加,提交的版本拿到了103.12;后来仔细思考,发现了这部分余地,仅增加了一行代码,分数便提高了不少;
- SAP同时支持DAG与DCG(有环图),仔细分析如下这个特殊情况可能有助思考:(给定节点1与8,求其SAP)
(Made with Graphviz) - BFS如由节点1先开始,则轮到第4次迭代时,节点1遍历,发现共同祖先节点5,路径距离为4+3=7,但此时不能停止搜索;实际SAP的共同祖先为1,路径距离为0+4=4,是在第4次迭代的节点8遍历部分被发现的。
public class BreadthFirstDirectedPaths {
private static final int INFINITY = Integer.MAX_VALUE;
private boolean[] marked; // marked[v] = is there an s->v path?
private int[] edgeTo; // edgeTo[v] = last edge on shortest s->v path
private int[] distTo; // distTo[v] = length of shortest s->v path
public BreadthFirstDirectedPaths(Digraph G, int s) {
marked = new boolean[G.V()];
distTo = new int[G.V()];
edgeTo = new int[G.V()];
for (int v = 0; v < G.V(); v++)
distTo[v] = INFINITY;
bfs(G, s);
}
public BreadthFirstDirectedPaths(Digraph G, Iterable<Integer> sources) {
marked = new boolean[G.V()];
distTo = new int[G.V()];
edgeTo = new int[G.V()];
for (int v = 0; v < G.V(); v++)
distTo[v] = INFINITY;
bfs(G, sources);
}
// BFS from single source
private void bfs(Digraph G, int s) {
Queue<Integer> q = new Queue<Integer>();
marked[s] = true;
distTo[s] = 0;
q.enqueue(s);
while (!q.isEmpty()) {
int v = q.dequeue();
for (int w : G.adj(v)) {
if (!marked[w]) {
edgeTo[w] = v;
distTo[w] = distTo[v] + 1;
marked[w] = true;
q.enqueue(w);
}
}
}
}
// BFS from multiple sources
private void bfs(Digraph G, Iterable<Integer> sources) {
Queue<Integer> q = new Queue<Integer>();
for (int s : sources) {
marked[s] = true;
distTo[s] = 0;
q.enqueue(s);
}
while (!q.isEmpty()) {
int v = q.dequeue();
for (int w : G.adj(v)) {
if (!marked[w]) {
edgeTo[w] = v;
distTo[w] = distTo[v] + 1;
marked[w] = true;
q.enqueue(w);
}
}
}
}
public boolean hasPathTo(int v) {
return marked[v];
}
public int distTo(int v) {
return distTo[v];
}
public Iterable<Integer> pathTo(int v) {
if (!hasPathTo(v)) return null;
Stack<Integer> path = new Stack<Integer>();
int x;
for (x = v; distTo[x] != 0; x = edgeTo[x])
path.push(x);
path.push(x);
return path;
}
}
public class SAP
{
// constructor takes a digraph (not necessarily a DAG)
private Digraph digraph;
public SAP(Digraph G)
{
this.digraph = new Digraph(G);
}
// length of shortest ancestral path between v and w; -1 if no such path
public int length(int v, int w)
{
if (!isValidVertex(v) || !isValidVertex(w))
throw new java.lang.IndexOutOfBoundsException();
int[] result = findAncestorAndDist(v, w);
return result[1];
}
//private String vertexRepresent(int
private boolean isValidVertex(int v)
{
return (v >= 0 && v < this.digraph.V());
}
// a common ancestor of v and w that participates in a shortest ancestral path;
//-1 if no such path
public int ancestor(int v, int w)
{
if (!isValidVertex(v) || !isValidVertex(w))
throw new java.lang.IndexOutOfBoundsException();
int[] result = findAncestorAndDist(v, w);
return result[0];
}
private int[] findAncestorAndDist(Iterable<Integer> v, Iterable<Integer> w)
{
BreadthFirstDirectedPaths bpdsV = new BreadthFirstDirectedPaths(digraph, v);
BreadthFirstDirectedPaths bpdsW = new BreadthFirstDirectedPaths(digraph, w);
int root = -1;
int distance = Integer.MAX_VALUE;
for (int i = 0; i < this.digraph.V(); i++)
{
if (bpdsV.hasPathTo(i) && bpdsW.hasPathTo(i) && bpdsV.distTo(i)
+ bpdsW.distTo(i) < distance)
{
root = i;
distance = bpdsV.distTo(i) + bpdsW.distTo(i);
}
}
if (distance == Integer.MAX_VALUE)
distance = -1;
int[] result = {root, distance};
return result;
}
private int[] findAncestorAndDist(int v, int w)
{
BreadthFirstDirectedPaths bpdsV = new BreadthFirstDirectedPaths(digraph, v);
BreadthFirstDirectedPaths bpdsW = new BreadthFirstDirectedPaths(digraph, w);
int root = -1;
int distance = Integer.MAX_VALUE;
for (int i = 0; i < this.digraph.V(); i++)
{
if (bpdsV.hasPathTo(i) && bpdsW.hasPathTo(i) && bpdsV.distTo(i)
+ bpdsW.distTo(i) < distance)
{
root = i;
distance = bpdsV.distTo(i) + bpdsW.distTo(i);
}
}
if (distance == Integer.MAX_VALUE)
distance = -1;
int[] result = {root, distance};
return result;
}
// length of shortest ancestral path between any vertex in v and any vertex in w;
//-1 if no such path
public int length(Iterable<Integer> v, Iterable<Integer> w)
{
int[] result = findAncestorAndDist(v, w);
return result[1];
}
// a common ancestor that participates in shortest ancestral path; -1 if no such path
public int ancestor(Iterable<Integer> v, Iterable<Integer> w)
{
for (int i :v)
{
if (!isValidVertex(i))
throw new java.lang.IndexOutOfBoundsException();
}
for (int i :w)
{
if (!isValidVertex(i))
throw new java.lang.IndexOutOfBoundsException();
}
int[] result = findAncestorAndDist(v, w);
return result[0];
}
}
private Map<Integer, Integer> getAncestors(int v) {
Queue<Integer> vQ = new Queue<Integer>();
Map<Integer, Integer> vM = new HashMap<Integer, Integer>();
vQ.enqueue(v);
vM.put(v, 0);
while (!vQ.isEmpty()) {
int head = vQ.dequeue();
int currentDist = vM.get(head);
for (Integer w : G.adj(head)) {
if (!vM.containsKey(w) || vM.get(w) > currentDist + 1) {
vQ.enqueue(w);
vM.put(w, currentDist + 1);
}
}
}
return vM;
}
// length of shortest ancestral path between v and w; -1 if no such path
public int length(int v, int w) {
if (v < 0 || v >= G.V())
throw new IndexOutOfBoundsException();
if (w < 0 || w >= G.V())
throw new IndexOutOfBoundsException();
Map<Integer, Integer> ancestorV = getAncestors(v);
Map<Integer, Integer> ancestorW = getAncestors(w);
int dist = -1;
for (Entry<Integer, Integer> items : ancestorV.entrySet()) {
if (ancestorW.containsKey(items.getKey())) {
int currentDist = ancestorW.get(items.getKey())
+ items.getValue();
if (dist < 0 || currentDist < dist)
dist = currentDist;
}
}
return dist;
}
// a common ancestor of v and w that participates in a shortest ancestral
// path; -1 if no such path
public int ancestor(int v, int w) {
Map<Integer, Integer> ancestorV = getAncestors(v);
Map<Integer, Integer> ancestorW = getAncestors(w);
if (v < 0 || v >= G.V())
throw new IndexOutOfBoundsException();
if (w < 0 || w >= G.V())
throw new IndexOutOfBoundsException();
int dist = -1, anc = -1;
for (Entry<Integer, Integer> items : ancestorV.entrySet()) {
if (ancestorW.containsKey(items.getKey())) {
int currentDist = ancestorW.get(items.getKey())
+ items.getValue();
if (dist < 0 || currentDist < dist) {
dist = currentDist;
anc = items.getKey();
}
}
}
return anc;
}
// length of shortest ancestral path between any vertex in v and any vertex
// in w; -1 if no such path
public int length(Iterable<Integer> v, Iterable<Integer> w)
throws IndexOutOfBoundsException {
int dist = -1;
for (Integer eV : v) {
for (Integer eW : w) {
int currentDist = length(eV, eW);
if (currentDist > 0 && (dist < 0 || currentDist < dist))
dist = currentDist;
}
}
return dist;
}
// a common ancestor that participates in shortest ancestral path; -1 if no
// such path
public int ancestor(Iterable<Integer> v, Iterable<Integer> w)
throws IndexOutOfBoundsException {
int dist = -1, anc = -1;
for (Integer eV : v) {
for (Integer eW : w) {
int currentDist = length(eV, eW);
if (currentDist > 0 && (dist < 0 || currentDist < dist)) {
dist = currentDist;
anc = ancestor(eV, eW);
}
}
}
return anc;
}
public class WordNet
{
//private Digraph digraph;
private Hashtable<String, Bag<Integer>> words;
private ArrayList<String> synsets;
private SAP sap;
//private int numSynsets
public WordNet(String synsets, String hypernyms)
{
this.words = new Hashtable<String, Bag<Integer>>();
this.synsets = new ArrayList<String>();
int numSynsets = parseSynsets(synsets);
Digraph digraph = parseHypernyms(hypernyms, numSynsets);
this.sap = new SAP(digraph);
}
private Digraph parseHypernyms(String hypernyms, int numSynsets)
{
Digraph digraph = new Digraph(numSynsets);
In hyper = new In(hypernyms);
while (!hyper.isEmpty())
{
String[] s = hyper.readLine().split(",");
//if (s.length < 2)
//throw new java.lang.IllegalArgumentException();
int from = Integer.parseInt(s[0]);
for (int i = 1; i < s.length; i++)
{
int to = Integer.parseInt(s[i]);
digraph.addEdge(from, to);
}
}
hyper.close();
// check if graph has a cycle
DirectedCycle dc = new DirectedCycle(digraph);
if (dc.hasCycle())
throw new java.lang.IllegalArgumentException("Cycle detected");
// check if one root
int roots = 0;
for (int v = 0; v < numSynsets; v++)
{
if (!digraph.adj(v).iterator().hasNext())
roots++;
}
if (roots > 1)
throw new java.lang.IllegalArgumentException("Multiple roots: "+roots);
return digraph;
}
private int parseSynsets(String synsetsFN)
{
int count = 0;
In syns = new In(synsetsFN);
while (!syns.isEmpty())
{
String[] s = syns.readLine().split(",");
if (s.length < 2)
throw new java.lang.IllegalArgumentException();
int id = Integer.parseInt(s[0]);
this.synsets.add(id, s[1]);
String[] wordsArray = s[1].split(" ");
for (String w: wordsArray)
{
Bag bag;
if (this.words.containsKey(w)) bag = this.words.get(w);
else bag = new Bag<Integer>();
bag.add(id);
this.words.put(w, bag);
}
count++;
}
syns.close();
return count;
}
// the set of nouns (no duplicates), returned as an Iterable
public Iterable<String> nouns()
{
return Collections.list(this.words.keys());
}
// is the word a WordNet noun?
public boolean isNoun(String word)
{
return this.words.containsKey(word);
}
// distance between nounA and nounB (defined below)
public int distance(String nounA, String nounB)
{
if (!isNoun(nounA) || !isNoun(nounB))
throw new java.lang.IllegalArgumentException();
Bag<Integer> v = this.words.get(nounA);
Bag<Integer> w = this.words.get(nounB);
return this.sap.length(v, w);
}
// a synset (second field of synsets.txt) that is
// the common ancestor of nounA and nounB
// in a shortest ancestral path (defined below)
public String sap(String nounA, String nounB)
{
if (!isNoun(nounA) || !isNoun(nounB))
throw new java.lang.IllegalArgumentException();
Bag<Integer> v = this.words.get(nounA);
Bag<Integer> w = this.words.get(nounB);
int root = this.sap.ancestor(v, w);
return this.synsets.get(root);
}
}