[vimeo 165483420 w=640 h=360]<\/p>\n
Introduction to OpenRefine<\/a> from Sarah Hatoum<\/a> on Vimeo<\/a>\u00a0(also available\u00a0on YouTube<\/a>). \u00a0In the video’s description, there are time ranges listed if you would like\u00a0to skip to different sections of the video.<\/p>\n This tutorial\u00a0(recorded using SnagIt) is intended to introduce users to OpenRefine, its basic features, and to act as a springboard for a digital humanities project\/study that involves great quantities of data. The dataset used in this Skillshare was generated from New York Public Library\u2019s (NYPL\u2019s) crowdsourcing project <\/span>What\u2019s on the Menu?<\/span><\/i><\/a>, where members of the public transcribed menu items from the 1840s to the present. Cleaning datasets is often the first step that needs to be taken when using\u00a0public datasets, as they are typically\u00a0messy, and OpenRefine can help users accomplish large-scale data cleaning (and data manipulation).<\/span><\/p>\n Overall, this\u00a0Skillshare intends to introduce users to\u00a0basic features of OpenRefine in order to make datasets more discernable (particularly for digital humanists who often use public humanities datasets), readying them for further analysis.<\/span><\/p>\n OpenRefine, formerly known as GoogleRefine and re-branded in 2012, is an open source software that can be used to clean, transform, and reconcile datasets. Publicly available raw datasets can be messy; if data is manually entered\u00a0into a spreadsheet, there is room for human error–for example, there can be several variations of a word due to typos, differences in capitalization conventions or trailing\/leading whitespaces (extra spaces after or before a word). Data can also be transformed (changed from one format to another) and reconciled (linked to data in external pages\/databases and cross-checked for accuracy).\u00a0<\/span><\/p>\n OpenRefine allows you see the \u201cbig picture\u201d of your data, interact with your data, and ask questions about your data–and sometimes answer those selfsame questions–quickly and fairly easily. <\/span><\/p>\n The OpenRefine Wiki offers \u201csmall workflows and code fragments\u201d called <\/span>recipes<\/span><\/a>. You can perform more complex transformations using these recipes; others have created these recipes to streamline your process of data cleaning and analysis. Use them!\u00a0<\/span><\/p>\n Reconciliation is another feature that can\u00a0be useful for information professionals (museum professionals, archivists, librarians) because it allows you to check the consistency of your data against data in\u00a0an external\u00a0database; you can check\u00a0your collection’s vocabulary with a controlled vocabulary. See the end of this post for reconciliation sources.<\/p>\n Lastly, you can add various extensions<\/a>\u00a0in order to add new functionalities to OpenRefine.<\/p>\n As mentioned, cleaning data is often the very first step that needs to be taken when\u00a0working with a public dataset. In a practicum at the Maryland Institute for Technology in the Humanities<\/a>, Lydia Zvyagintseva and Trevor Mu\u00f1oz imagined ways to curate the dataset generated from NYPL\u2019s <\/span>What\u2019s on the Menu?<\/span><\/i>\u00a0project and make it more digestible for digital humanities researchers. Currently, as Zvyagintseva noted, there is not a thematic\/categorical classification of the 24,000 digitized menus, so she and Mu\u00f1oz aimed to determine ways to classify (curate) the collection.<\/span><\/p>\n After assessing all data, they received summaries of requests from researchers who used NYPL\u2019s <\/span>What\u2019s on the Menu?<\/span><\/i> API. This helped Zvyagintseva and Mu\u00f1oz understand user needs (regarding the use of data), and determine the types of questions humanities researchers would ask of the dataset(s).\u00a0In order to classify the datasets accordingly, they first needed to use OpenRefine:<\/span><\/p>\n We used OpenRefine…software to cluster and rename data fields that we considered of particular use or interest to future users of this collection, principally names of the businesses offering these menus and also, where present, the names of the categories (supplied by original cataloguers?) to which the menus had been assigned. While OpenRefine helped with the initial clustering, the large number of name and spelling variations meant some tedious line-by-line editing. This is also the data curator\u2019s job.<\/p>\n<\/blockquote>\n After using OpenRefine to clean up humanities data similar to the <\/span>What\u2019s on the Menu? <\/span><\/i>datasets, questions can be asked from various disciplinary perspectives and can ultimately guide how a humanist develops his\/her DH project.<\/span>\u00a0<\/b><\/p>\n Besides Microsoft Excel, <\/span>Data Wrangler<\/b><\/a> can be used as an alternative to OpenRefine. Data Wrangler is an open source, web-based application (as opposed to OpenRefine, which is desktop-based) for manipulating data. Data Wrangler allows for \u201c…i<\/span>nteractive transformation of messy, real-world data into the data tables analysis tools expect. Export data for use in Excel, R, Tableau, Protovis, [etc.]\u201d While OpenRefine may be\u00a0best for cleaning and reconciling data, Data Wrangler may be best used for reorganizing or reformatting data.<\/span><\/p>\n Note:<\/strong> *A novice OpenRefine user expressed uncertainty with the definition of facet, and the definition has been updated. However, if you would like to learn more, I would recommend reading\u00a0through OpenRefine’s Wiki page for faceting\/filtering.<\/a><\/p>\n References and further reading:<\/b><\/p>\n Download OpenRefine. (2013). Retrieved from\u00a0http:\/\/openrefine.org\/download.html<\/a><\/p>\n Hooland, S. V., Verborg, R., & Wilde, M. D. (2013, August 5). Cleaning data with OpenRefine. <\/span>Programming Historian<\/span><\/i>. Retrieved from <\/span>http:\/\/programminghistorian.org\/lessons\/cleaning-data-with-openrefine<\/span><\/a><\/p>\n Huynh, D. (2011, Feb). Google Refine. Retrieved from <\/span>http:\/\/davidhuynh.net\/spaces\/nicar2011\/tutorial.pdf<\/span><\/a><\/p>\n Morris, T. General Refine Expression Language. (2015 May 19). Retrieved from\u00a0https:\/\/github.com\/OpenRefine\/OpenRefine\/wiki\/General-Refine-Expression-Language<\/a><\/p>\n Mu\u00f1oz, T. (2013, Aug 19). Refining the problem \u2014 More work with NYPL’s open data, part two. Retrieved from http:\/\/www.trevormunoz.com\/notebook\/2013\/08\/19\/refining-the-problem-more-work-with-nypl-open-data-part-two.html<\/a><\/span><\/p>\n Mu\u00f1oz, T. & Rawson, K. (2014). Curating menus. Retrieved from curatingmenus.org<\/a><\/span><\/p>\n [tfmorris]. (2012, Dec 22). Screencasts: Screencasts about Refine. Retrieved from\u00a0https:\/\/github.com\/OpenRefine\/OpenRefine\/wiki\/Screencasts<\/a><\/p>\n Verborgh, R. & Wilde, M. D. (2013, Sept). Analyzing and fixing data. In <\/span>Using OpenRefine<\/span><\/i>. Packt Publishing. Retrieved from <\/span>https:\/\/www.packtpub.com\/sites\/default\/files\/9781783289080_Chapter_02.pdf<\/a>\u00a0<\/span>[*An e-book version can be found <\/span>HERE<\/span><\/a>.]<\/span><\/p>\n Zvyagintseva, L. (2013, Jun 21). Organizing historical menus: a data curation experiment. MITH. Retrieved from <\/span>http:\/\/mith.umd.edu\/taxonomizing-historical-menus-a-data-curation-project\/<\/span><\/a><\/p>\n <\/b>Reconciliation resources:<\/strong><\/p>\n [alctsce]. (2014 Mar 3). Using OpenRefine to update, clean up, and link your metadata to the wider world. [Video file]. Retrieved from\u00a0<\/span>https:\/\/www.youtube.com\/watch?v=E-NbMR3_MRw&list=UUvPEK3a3Qb0GMCSzqoZDnlg<\/span><\/a><\/p>\n Heller, M. (2013 May 1). A librarian’s guide to OpenRefine.<\/span> ACRL TechConnec<\/span><\/i>t. Retrieved from <\/span>http:\/\/acrl.ala.org\/techconnect\/post\/a-librarians-guide-to-openrefine<\/span><\/a><\/p>\n Multimedia Lab, MasTIC, & Information School at the University of Washington. (2016). Free your metadata. Retrieved from freeyourmetadata.org<\/a>.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":" \n This tutorial\u00a0(recorded using SnagIt) is intended to introduce users to OpenRefine, its basic features, and to act as a springboard for a digital humanities project\/study that involves great quantities of data. The dataset used in this Skillshare was generated from New York Public Library\u2019s (NYPL\u2019s) crowdsourcing project <\/span>What\u2019s on the Menu?<\/span><\/i><\/a>, where members of the public transcribed menu items from the 1840s to the present. Cleaning datasets is often the first step that needs to be taken when using public datasets, as they are typically messy, and OpenRefine can help users accomplish large-scale data cleaning (and data manipulation). <\/span><\/p>\n Overall, this\u00a0Skillshare intends to introduce users to\u00a0basic features of OpenRefine in order to make datasets more discernable (particularly for digital humanists who often use public humanities datasets), readying them for further analysis.<\/span><\/p>\n<\/p>\nAbout the Skillshare:<\/h2>\n
What is OpenRefine?<\/h2>\n
What will I learn in the Skillshare video?<\/h2>\n
\n
OpenRefine features:<\/h4>\n
\n
\n
\n
\n
\n
\n
What are some advanced features of OpenRefine?<\/h2>\n
How can OpenRefine be used as a digital humanities tool?<\/h2>\n
\n
What is an alternative to OpenRefine?<\/h2>\n
\n