Company of choice – WePay
Json file already created
As described in the first assignment, the term project involves the creation of a knowledge graph for competitive intelligence. The knowledge graph will be partially curated, and generated from files that we create manually. These files are supplemented by data from documents that we gather programmatically by crawling and scraping the web. In this assignment, you are asked to summarize the work that has been completed through week 5. Identify the sources of data and methods of generating and acquiring those data. Methods should include unstructured text data from a focused crawler, as well as initial structured data that you have created. There may also be structured data from APIs.
Nickel et al. (2015) list four general approaches for constructing knowledge bases, where the foundational elements are triples (noun/subject, predicate, noun/object). Which of these four approaches do you plan to use in constructing your knowledge graph?
Much of these data should reside in JSON lines files. with thousands of JSON objects. Ideally, the body of text from the focused crawler consists of meaningful English paragraphs and sentences. (Note that social media data may be useful as a foundation for assessing buyer opinions in competitive intelligence research.)
There are many standard JSON formats to draw on as shown at https://schema.org/ (Links to an external site.) For example, here is a schema for an organization: https://schema.org/Organization (Links to an external site.)
The report should also address your plans for storing these data in a database such as PostgreSQL. To what extent do you plan to utilize relational (table-oriented), document (JSON), geometry data types (latitude and longitude), and graph data capabilities of the database?
This page should help to guide your thinking about knowledge graphs for competitive intelligence: Knowledge Graph Schema
Report Components
Organize the paper as a research report with each section answering basic questions as identified in bold type below:
Abstract. An executive summary of the research.
Introduction. Why are you conducting this research?
Literature review. Who else has conducted research like this?
Methods. How are you conducting the research?
Results. What did you learn from the research?
Conclusions. So, what does it all mean?
When drawing conclusions, make sure that you relate your study results back to the management problem you have identified. To what extent will the collected data be useful in addressing the management problem (the creation of a knowledge graph for competitive intelligence)?
See Structure of Formal Research Reports Download Structure of Formal Research Reports for more detail.
Grading Guidelines
Data preparation, exploration, visualization (20 points)
Review research design and modeling methods (20 points)
Review results, evaluate models (20 points)
Implementation and programming (20 points)
Exposition (20 points). The paper should include a problem descriiption and management recommendations. Discuss each of the eight activities associated with the development of a focused crawler a la Chakrabarti, van den Berg, and Dom (1999).
Deliverables and File Formats
Create a folder or directory with all supplementary files with your last name at the beginning of the folder name, compress that folder with zip compression, and post the zip-archived folder under the assignment link in Canvas. (Use zip, not StuffIt or any 7z or any other compression method.)
The paper should consist of no more than four pages of text, double-spaced. Figures and tables may go in an appendix and do not count as part of the four pages. The paper must be provided as an Adobe Acrobat pdf file. (MS Word files are NOT acceptable.)
Program files should be provided as plain text. Go and Python are the recommended programming languages. These may be supplemented by plain text files showing SQL and OpenCypher database queries.
It is recommended that you begin with the jump-start code provided by the instructor.
Output from programs, such as console listings/logs, text files, and graphics output for visualizations. The collection of documents from the crawl is a key deliverable of this assignment and should be included in the submitted zip archive as a JSON lines file.
List of file names and descriiptions of files in the zip-compressed folder/directory. Recognizing that long file names can pose problems on systems that have limited path lengths, ensure that file names are short but descriiptive. For example, the file for the paper could have the form lastname-firstinitial-paper.pdf as in miller-t-paper.pdf
Reference
Nickel, Maximillian, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A Review of Relational Machine Learning for Knowledge Graphs. Proceedings of the IEEE, 104(1): 11–33. Online reference: https://arxiv.org/abs/1503.00759 (Links to an external site.) Paper: https://arxiv.org/pdf/1503.00759.pdf (Links to an external site.)
